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ABSTRACT 

This report provides an overview of the Oklahoma 
School Testing Program (OSTP) and summarizes students' average 
writing assessment scores from 1987 through 1992* Oklahoma has 
completed its 6th year of administering a state legislature mandated 
test to 10th graders and its 5th year of testing 7th graders. The 
Stanford Writing Assessment Program was adopted in 1992. Over 38,500 
7th graders and over 32,600 10th graders were tested. Student papers 
were hand scored through holistic and analytic scoring techniques 
designed to measure different aspects of composition skills. 
Comparisons of the scores of Oklahoma students with the national 
average during the years such tests have been administered indicates 
continued above-average performance for both grades. With the change 
to the Stanford tests in 1992, 7th graders showed an apparent 
increase in score differential, while 10th graders showed an apparent 
decrease. The shift to the new test may account for differences in 
comparisons with national norms for 1992 and previous years. 
Information for understanding and using the test results is 
presented. Seven tables and eight figures present test results and 
trend information. Five appendixes contain an additional eight tables 
that give inf ormat ion about s tudent performance , scores under the 
normal curve, and averages at the district level for both grades. 
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To the Citizens of Oklahoma: 

The following report provides an overview of the Oklahoma School Testing Program (OSTP) and 
summarizes students' average writing assessment scores from 1987 through 1992. The Writing 
Assessment Component of the Oklahoma School Testing Program was implemented in compliance 
with Title 70 O.S. Supp. 1985, Section 1210.510 et. seq. (Senate Bill 426, enacted in 1986). 

Oklahoma has completed its sixth consecutive year of administering a direct writing assessment 
to students in tenth grade, and our fifth year of administering such a test to seventh-grade students. 
The State Board of Education adopted the Stanford Writing Assessment Program (published by the 
Psychological Corporation) for use in 1992. 

Oklahoma law requires that inservice training be provided for educators who administer tests 
within the OSTP. The State Department of Education provides pretest and posttest inservices for all 
school districts. The purpose of the pretest inservice is to prepare educators to administer the tests 
according to the procedures used in the standardization and to utilize all appropriate security and 
accuracy guidelines in handling test materials. Posttest inservice sessions focus on instructing 
educators in the appropriate interpretation and use of test scores to improve instructional programs 
(intent of Senate Bill 426). It is my belief that educators are working dilligently to utilize test results 
in this manner. 

Oklahoma's 1992 Stanford Writing Assessment scores indicate that Oklahoma students are 
continuing to perform at levels above the national average. I commend Oklahoma's teachers and 
administrators for providing quality writing instructional programs. 

The State Department of Education appreciates the cooperation of all public school educators in 
the local management of the OSTP. In addition, we appreciate parents' assistance in preparing Uieir 
children mentally, physically and emotionally for participation in this program. It is only with such a 
spirit of school and community cooperation that we can successfully obtain a measure of our students' 
best efforts to reveal their accumulated knowledge and academic skills. 



Sincerely, 




Sandy Gar rett 
State Superintendent 



2600 N. Lincoln Blvd., Oklahoma City, OK 73106-4699 (406) 621-3301, Fax: (406) 621-6205 

First By The Twenty- First 



OKLAHOMA SCHOOL TESTING PROGRAM (OSTP) 
WRITING ASSESSMENT COMPONENT: 

EXECUTIVE SUMMARY- 1992 

The Stanford Writing Assessment was administered to Oklahoma's seventh- and tenth-grade students 
in February, 1992. Students' papers were hand-scored through use of two different procedures- 
holistic and analytic scoring. These procedures are designed to measure different aspects of 
composition skills. By using the combination of the two scoring procedures, we are supplying teachers 
with a variety of performance data that can be used to assess students' writing skill strengths, 
determine areas of skill reinforcement need, and provide focus for the most beneficial instructional 
methods for each student. 

Holistic scores provide an overview of the general quality of composition skills demonstrated by each 
student in response to a verbal writing prompt. Holistic Raw Scores range from 2 to 12 and are 
converted to normative scores to provide a comparison of student performance with that of the 
national norm group. 

The average Holistic- Score Percentile Ranks for Grades 7 and 10 were 63 and 57, respectively. A 
Percentile Rank of 50 is considered the "national average." A score of 63, for example, means that the 
average seventh-grade student in Oklahoma scored as well as or better than 63% of the students in 
the national norm group who were administered this Game prompt. 

Analytic Scores are given on a scale of one to four, with four being the highest performance level. The 
analytic scoring procedure for 1992 was expanded to evaluate the six specific areas of writing skills 
listed below. (Figure 1, page 16 provides further delineation of these skills). The following table 
describes Oklahoma's 1992 performance in terms of the skill area, performance level within each skill 
area, and the percent of students scoring in each of the performance levels. 



ANALYTIC SCORE SUMMARY--1992 



Grade 
10 





I 

Ideas and 
Development 


II 

Organization, 
Unity ♦ and 
Coherence 


ill 

Word 

Choice 


IV 

Sentences and 
Paragraphs 


V 

Grammar and 
Usage 


VI 

Mechanics 
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% of Students 
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4 (high) 
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50 
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4 (high) 
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25 
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58 


51 


62 


47 


47 


45 
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23 


33 


23 


29 


23 


24 


1 (low) 


2 


3 


2 


6 


4 


6 



Comparisons of Oklahoma students' writing scores with the national average during the years such 
tests have been administered indicates continued above-average performance at both grades tested. 
The change to the new Stanford Assessment from the MATS Writing Test indicates that, compared 
with national norm group performance on the two tests, Oklahoma's 7th grade students show an 
apparent increase in score differential, while the 10th grade scores indicate an apparent decrease, The 
shift f .o the new test with different writing and scoring procedures-along with an increased national 
level of writing achievement, especially at the high school grades (as indicated by the norms)~are 
possible reasons for the differences seen in the 1992 and previous Oklahoma writing assessment 
performance comparisons with national norms. 
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SUMMARY REPORT FOR THE 
OKLAHOMA SCHOOLTESTING PROGRAM (OSTP) 

WRITING ASSESSMENT COMPONENT - 1992 

Introduction 

Beginning in 1986, Oklahoma Law mandated that a standardized, norm-referenced direct 
writing assessment be administered to students in specified grades. This program-referred 
to as the Writing Assessment Component of the OSTP--began with administration of writing 
tests to students in grade 10 in 1987. The program was expanded in 1988 to include 
assessment at grade 7. 

The measurement instruments utilized to achieve an assessment of students' writing skills 
are obtained through the state bid system. Such instruments achieve this measure by 
providing a writing prompt to which students respond with an essay. Students' writing is 
hand-scored by three professional raters who apply two different procedures with each 
paper— holistic scoring and analytic scoring. Use of the former method results in a judgment 
of the paper's overall merit as compared with exemplary papers written during the national 
standardization of the tests, and yields the normative data required by law. Employment of 
the analytic scoring method yields performance estimates on six areas of specific writing 
skills. Both scoring methodologies are fully explained later in this report. 

The measurement instruments adopted by the State Board of Education for this program 
include the MATS Writing Test (1987-1991) and the Stanford Writing Assessment Program 
(1992), both of which are published by The Psychological Corporation. While both test series 
assess students' actual writing skills, they differ in their philosophical bases and resulting 
methodologies. These differences will be discussed at length in a subsequent section of this 
report. 

The following report describes the inception, implementation procedures, assessment 
instruments and results of this program. District level data are reported for the 1991-92 
assessment, and state level data are provided for the writing assessments conducted in 1987 
through 1992. In addition, guidelines are provided for analysis and interpretation of the test 
results, 

A. Inception of the OSTP and the Writing Assessment Component: Legislation 

Oklahoma House Bill 1480, enacted in 1984, required the State Department of 
Education to develop a plan for statewide assessment of student achievement in the basic 
skills. This plan was submitted to the Legislature on January 1, 1985. In 1985, the 
Oklahoma School Testing Program Act (§ 1210.505 through 1210.511 of Title 70 of the 
Oklahoma Statutes) mandated the administration of a standardized, norm-referenced 
achievement test to students in grades 3, 7, and 10 (House Bill 1466). The subject and 
skill areas measured in this assessment include reading, language arts, mathematics, 
science, and social studies. Data resulting from the OSTP are reported annually to the 
Oklahoma Legislature and the State Board of Education. 
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The Oklahoma School Testing Program Act further required the State Department of 
Education to submit a plan for administering a direct writing assessment to Oklahoma's 
public school students, Submitted to the Oklahoma Legislature on January 1, 1986, this 
plan addressed (1) definitions of the various components of the writing assessment; (2) 
estimated costs; (3) timelines and procedures for implementation; (4) suggested tests to 
be utilized to provide consistent data; (5) reporting procedures; and (6) anticipated 
benefits to be achieved through the program. 

Oklahoma Senate Bill 426 (1986) created the Writing Assessment Component of the 
OSTP, mandating that a standardized, norm-referenced direct writing test be 
administered to tenth-grade students during the 1986-87 school year. This law further 
stated that, based on the availability of funds, the program would be expanded in 1987- 
88 to include grade 7 and expanded again in 1988-89 to include grade 3. (Subsequent 
legislation [i.e, Senate Bill 183, enacted in 1989] deleted any reference to adding grade 3; 
therefore, the program has been maintained at grades 7 and 10 only.) 

Based on Senate Bill 426, the purpose of the writing assessment is to obtain a measure of 
students' actual writing performances; its function is to achieve improvement in the 
state public education system. Furthermore, "the State Board of Education shall seek to 
ensure that data yielded from the test is utilized at the school district level to prescribe 
skill reinforcement and/or remediation by remuring school districts to develop and 
implement a specific program of improvement based on the test results" (70 O.S. Supp. , 
§ 1210.508). In addition, this bill stated that "the State Department of Education shall 
provide inservice training for local school personnel who administer the test." 

The legislation required that reports of test results be provided to local school districts by 
the contracting test company. Individualized results of the test "shall be made available 
by the local school district to the classroom teachers in the area tested" ...and "the 
Department will provide test results to appropriate public officials as required by law." 

Oklahoma House Bill 1110 (1987) expanded the Writing Assessment Component of the 
OSTP-with the same requirements--to include administration of a direct writing test to 
both seventh- and tenth-grade students. Subsequent legislation has maintained the 
current program. 

Senate Bill 426 and House Bill 1110 further stated that students who have 
Individualized Education Plans (IEPs) pursuant to Public Law 94-142 shall not be 
required to participate in the Oklahoma School Testing Program. Due to effects of 
Federal Law (Section 504 of the Rehabilitation Act of 1973, 29 U.S.C. 706, regarding the 
prevention of discrimination against the handicapped), the Stato. Department of 
Education issued the following procedural directive to all public school superintendents 
and dependent school principals annually through the school year 1988-89: 

"Based on Oklahoma Law, students who have Individualized Education Plans (IEPs) 
pursuant to Public Law 94-142 shall not be required to participate in the Oklahoma 
School Testing Program. However, the right of these students to participate in the 
OSTP cannot be denied. Parents must be given the opportunity to sign a form 
indicating their desire to have their children take the tests. The form to be used is 
the TCP Program SDE Form 9' which is employed during the IEP Conference with 
parents. Please make sure that parents respond to the question on this form 
regarding their children's participation in the OSTP." 
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Senate Bill 183 (Section 1210.511 B.) ( enacted in May of 1989, contained the following 
change in language regarding testing students with IEPs: 

* i student whose education is subject to the provisions of an individualized 
education plan (IEP) pursuant to Public Law 94-142, as amended, shall be 
tested pursuant to the provisions of the Oklahoma School Testing Program 
Act, Section 1210.505 et seq, of this title, only to the extent specified by the 
student's individualized education plan" 

With this change, the decision to test such students became a function of the IEP Team, 
of which the student's parent is a standing member The "IEP Program SDE Form 9" 
continued to be utilized as the documentation of the team's decision in regard to whether 
or not the student would be administered the tests. 

Regulations were adopted by the State Board of Education in 1987 and 1989 to specify 
test exemptions for students who have IEPs and those who are considered as Limited 
English Proficient (LEP) based on the definition issued by the U.S. Secretary of 
Education. (A summary of the OSTP Regulations is found on pages 8 and 9 of this 
report.) According to Regulation lb, students who are considered as LEP can be 
exempted from OSTP participation at local district option (1) if such students meet the 
definition; (2) if these students have been counted as LEP for accreditation reporting 
purposes at the time that tests are administered; and (3) if parental permission for 
exemption from testing is obtained in writing and on file in the school. 

Combining Assessment Methodologies in the OSTP 

Two methods are available to measure writing and related skills--/nc//rect and direct. 
Implementation of the OSTP Writing Assessment Component benefits teachers and 
students by providing data from both kinds of measurement. 

Indirect measurement is conducted through the administration of multiple-choice test 
items which consist of correct and incorrect optional answers illustrating the application 
of writing-related skills (e.g., word usage, sentence structure, punctuation, 
capitalization, grammar, and spelling). In such an assessment, the student must read 
the test item, decide which of the options is correct, and mark a machine-scorable answer 
blank to indicate his or her choice. The resulting scores do not indicate the student's 
ability to write; instead, indicated is the student's ability to apply knowledge of writing- 
related skills to printed sentences and phrases and choose the correct option. Students' 
strengths and deficiencies in skill knowledge/application are indicated by the number of 
correct choices made in the measurement of each skill. Such data are reported on the 
language achievement tests administered as part of the OSTP. 

Direct writing assessment is conducted through the provision of a stimulus or prompt to 
which the student resj " ds by writing a story , description, explanation, or other form of 
discourse designed to aduress the topic suggested by the prompt. Students' compositions 
are rated by professional graders. Rating of students' papers is based on predetermined 
criteria which represent specific writing skills and the level with which each is applied 
in the discourse, The direct assessment, therefore, provides an illustration of the 
student's actual ability to express his or her thoughts while demonstrating writing 
skills. 
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The major purpose of the OSTP is to provide data with which to determine the status of 
students' skill achievement levels. Its function is "to improve all phases of the state 
public educational system" (House Bill 1466). It is believed that tandem use of results 
from both components of the OSTP will ultimately function to increase the potential for 
improvement in public school writing programs. 

Use of the Standardized Writing Assessment Instrument: Benefits and Limitations 

A standardized writing test is one which has been constructed in accordance with 
detailed specifications as to kind and difficulty of prompt, time limits needed to complete 
the written response, and criteria utilized for scoring. In developing such tests, prompts 
are carefully constructed, field-tested with random samplings of students nationwide, 
and determined to be appropriate in difficulty and discriminating power. The data 
resulting from standardization of such tests are used to establish a national "norm" or 
average against which students' scores at the individual, class, building, and state levels 
can be compared. Analysis of test data at each of these levels reveal strengths and 
deficiencies in students' skills, thus providing a basis for planning instructional 
improvement applied to individuals as well as groups. 

I. Benefits 

The benefits derived from use of the standardized writing test include the following: 

• A special phenomenon transpires in communities and schools when use of 
standardized tests expands from the local, district-controlled level to that of 
statewide assessment. In regard to writing skills-the last of the "basic skills" to 
be assessed with standardized instruments-there is a renewed interest in (a) the 
skills being assessed, (b) the status of students' writing performance levels; (c) 
the need and desire for improvement; and (d) the methodology needed to achieve 
desired instructional goals. This interest -pursued with positive action-results 
in renewed motivation of teachers and students, more appropriate instruction at 
all relevant grade levels, initiation of interim evaluation at the classroom level, 
and subsequent progress achieved with improvement of students' skills. 

• The standardized writing test is administered using the writing process method 
within a timed format, thus providing an estimate of the student's ability to 
organize and record thoughts in response to a specific stimulus. Considered as 
one of the "survival" skills, writing is applied in many situations throughout an 
individual's life. Such applications include written class assignments for which 
time is limited; teacher-constructed essay tests; job applications which require a 
timed impromptu autobiography, discussion of job-related goals, and/or 
description of leisure-time pursuits; and the on-the-job creation of 
correspondence, reports, and other business communications. Use of the 
standardized writing test and its results can reawaken the educator's awareness 
of the student's need for success in such realistic applications. Reorientation to 
this factor often encourages educators to refocus instructional programs on 
students' specific skill needs, thus increasing the potential for long-range 
program improvement. 

• Use of the standardized writing test encourages educators to provide test- 
preparation sessions for students— a factor which helps individuals become more 
comfortable with test-taking. As teachers (a) help students learn to budget time; 
(b) counsel individuals privately on test results and their meaning for current 
needs and future success; and (c) reassure students through sharing progress 



4 13 



reports periodically, each is motivated to take ownership in the testing situation 
and its outcomes. As students assume greater responsibility for their own 
learning, their efforts to improve the quality of their performance begin to 
increase-a factor which contributes to a more comfortable testing situation. 

Limitations 

Due .to the structure of the standardized test and the specifications within which 
each is constructed, administered, and scored, there are certain limitations which 
need to be considered in the analysis and use of resulting data: 

• Standardized test results represent estimates of the student's demonstrable 
achievement level only at the time the test was given and for a few weeks 
thereafter. With the passage of time, accumulation of additional knowledge, and 
practice in writing, students' test scores can improve to a measurable degree. 
•Therefore, neither individual students, school buildings, nor school districts 
should be labeled nor judged on a single test administration. 

• Failure to control certain necessary standardized test conditions can yield 
invalid results. Conditions which need to be controlled during test 
administration include (a) physical environment (room temperature, lighting, 
noise level, and overcrowding of students); (b) emotional environment 
(diminishing test anxiety and motivating students through positive attitudes on 
the part of parents, teachers, and school administrators); and (c) test 
administration procedures (giving directions accurately, observing strict timing 
of tests, and monitoring students' test-taking behaviors). Appropriate 
precautions regarding the control of all conditions mentioned above are 
discussed in the Pretest Inservice Training Sessions (and manual) provided for 
all test administrators prior to testing. 

• It is neither equitable, meaningful, nor appropriate to compare the performance 
of individual schools nor school districts with one another without considering 
certain demographic factors which are known to exert influence on test results 
(Jenks, 1972; White, 1976). These factors include socioeconomic level of the 
community, median income, and median educational levels of community 
members. In addition, where small numbers of children are tested, the abilities 
of one or two students can affect the average score of any grade level of students 
in that district. 

Furthermore, the transient nature of our society can have an effect on the overall 
performance level of the school district. This factor-often referred to as student 
mobility rate, or population turnover rate-tends to exert a greater influence on 
scores of schools which enroll smaller numbers of students. 

In general, the higher the socioeconomic level of the community, the greater the 
n.edian income (factors which correlate with educational level); and the higher 
the median educational level of the community is, the more likely the students in 
that school district are to score higher on tests. 
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Within the time limitation imposed on the reporting of the enclosed results, the 
test data contained in this document could not be treated statistically to equalize 
the effects of those factors listed above. Under these conditions, it is both 
meaningless and inappropriate to compare the performance of students in 
specific schools or school districts to determine the effects of instructional 
programs. 

Knowledge of test-taking skills can enhance the student's ability to perform on 
standardized tests. Such skills affecting the direct writing assessment include 
practice in responding to a writing prompt under timed conditions, learning to 
organize the response before beginning to write, and editing/revising a piece of 
discourse either during the writing or immediately afterward. 

Current instructional theory advocates the use of the 'process-writing" approach 
which includes the following steps: (1) pre-writing (organization and outlining of 
id^as); (2) writing the draft; (3) revising and editing; and (4) publishing. A 
search of the literature on writing instruction reveals that, for the past 15 to 20 
years, this method has been proven to be a highly successful approach. 

The Stanford Writing Assessment Program, (Second Edition), adopted for 
Oklahoma's 1992 assessment, is designed to encourage students to utilize 
"process-writing" skills during the examination. Since this is a series of 
standardized, norm-referenced writing assessments, the tests are timed. 
Following directions in the manual, the test administrator guides students 
through a schedule which includes five minutes for the pre-writing stage; 20 
minutes for the writing stage; and the last 10 minutes for the revising and 
editing stage. 

While the test administrator provides oral directions to guide students through 
the timing of these three writing stages, students may elect to use the 35 minutes 
allowed for the test to construct the essay according to their own schedule. 
Performing well on such an assessment is often largely dependent on the degree 
to which the student has learned the "process-writing" skills and can apply them 
in this context. 

Teachers are encouraged to base daily instruction on the "process-writing" 
method, intensifying such instruction to the point of successful writing skill 
application by students. In addition, teachers are encouraged to provide frequent 
practice for students in prompt-initiated, timed writing sessions. The use of such 
methods-coupled with encouragement of an "ownership" attitude in students 
regarding writing practice and writing assessment-should serve to increase 
students' writing proficiencies. 



Implementation and Maintenance Procedures (1986-1992) 

A. Program Objectives 

Objectives for accomplishing the successful implementation of the Writing Assessment 
Component included the following: 

• To ohtain a test which meets the specifications outlined by Oklahoma law. 

• To provide a system whereby tests remain secure. 

• To provide pretest inservice training for all test administrators as outlined in the 
la*'. 

• To provide a system for dissemination, return, and accounting of test materials. 

• To provide a system for rating students' compositions— one which yields both 
normative data as well as data with which students' specific skill strengths and 
deficiencies can be determined-and for reporting of results to local schools. 

• To analyze, interpret, and report statewide results to the State Board of Education 
and the Legislature. 

• To provide posttest inservice training on the interpretation and use of test results at 
the local school level. 

• To assist school personnel in incorporating the use of test scores in their School 
Improvement Plans. 

B, Procedures Utilized to Accomplish the OSTP Objectives 

1. Obtaining a Standardized, Norm-Referenced Direct Writing Test through the State 
Bid System 

• In compliance with Oklahoma Law, a Request for Proposals (RFP) was developed 
and released to test publishers nationwide. This document provides a full 
description of the review and selection process to be employed; the major 
products and services for which bids are being requested; timelines for delivery of 
materials and services; scoring systems which would provide the kind of data 
needed to enhance instructional improvement; and all other attachments 
required by law. 

• A "mandatory attendance" Pre-Bid Conference is held within two weeks after 
release of the RFP. All bidders' questions received prior to this conference and 
the Department's responses are submitted in writing to participants in this 
conference. All requirements of the Writing Assessment Component of the 
OSTP are discussed and bidders are allowed to clarify any misconceptions 
regarding bid requirements. 

• The test review process is designed to obtain an objective evaluation of all 
companies' bids for products and services in terms of the manner in which the 
following requirements either the Department's or those of the law-will be met: 
( 1) test specifications; (2) scoring criteria and methodologies utilized; (3) security 
of test materials and individual students' score reports; (3) SDE staff training far 



7 



pretest and posttest inservice programs; (4) provision of inservice materials (i.e., 
in terms of quality, quantity, and deadlines); and (5) customization of formats 
designed for testing and reporting in a manner that best meets the need for 
instructional program improvement. 

Evaluation forms consisting of more than 100 criteria are utilized by review/evaluation 
committees to judge the appropriateness of products and services bid by companies. One 
form is completed by each reviewer to indicate his or her evaluation of each company's 
set of proposals and test instruments submitted. 

For evaluation of writing test bids, more than thirty (30) junior high and high school 
composition teachers, counselors, school administrators, testing experts from public 
schools, and university professors having expertise in either tests and measurements 
and/or writing instruction serve as the evaluation committee. During the evaluation 
process, committee members spend at least 2£ hours studying the proposals and tests bid, 
after which company representatives conduct two-hour presentations that include 
answering all questions raised by reviewers. 

One of the major factors the review committee is asked to consider is the degree to which 
the tests submitted by bidders measure Oklahoma's Suggested Learner Outcomes 
(SLOs) for composition. Considered as equally important is the degree to which scoring 
criteria and techniques (1) reflect measurement of the SLOs, and (2) provide information 
with which teachers can determine and remediate students' specific writing skill 
deficiencies. (Section III-C of this report [Figures 2 & 3, page 19-20] describes the match 
of measurement techniques with Oklahoma's SLOs and the manner in which such 
techniques produce results which relate to each of the measurable Learner Outcomes. 
Section III-C of this report describes the specific scoring criteria used to rate Oklahoma's 
students' compositions [holistic scoring, pages 12-15; analytic ^coring, pages 16-17.) 

Responses on the evaluation forms are tallied, revealing the committee's 
recommendation. The State Board of Education considers all proposals, studies the 
evaluation committee's recommendation, and adopts the appropriate program. 

2. Providing for Maintenance of Test Security 

Maintenance of test security has become a major issue-both nationally and in 
Oklahoma. The test security measures previously outlined in State Board 
Regulations were greatly enhanced during the 1989-90 school year. Additional 
security measures implemented in 1989-90 and maintained throughout 1992 are 
outlined as follows: 

• The test form being utilized is embargoed by the test publisher so that no person 
living in Oklahoma can obtain a copy other than through standard OSTP 
distribution at the time of testing. 

• Test booklets are delivered to schools in s .led envelopes which are to be opened 
at the beginning of the test session only by students who are to be administered 
the test. All unused tests are to remain in sealed envelopes. 

• Test materials are to remain locked in a secure area until the morning of the first 
day of test administration. 

• An adult monitor other than the test administrator is required to witness the test 
administration and securing of test materials. Selection of monitors is subject to 
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the approval of the district superintendent, dependent school principal, or 
building principal. 

• Security forms are provided for school superintendents, principals, building and 
district test coordinators, test administrators, and monitors. These forms state 
that, to the best of the participants' knowledge, the specified test security 
measures were followed. All forms are to be signed by the appropriate OSTP 
participants and returned with answer documents. An accounting of these 
signed forms is documented by the State Department of Education. 

• Adult participants in the OSTP are directed to refrain from overtly or covertly 
exposing students to the test items and/or answers to test items. 

• Violations of test security are to be reported to the State Department of 
Education, investigated, and proven incidents can result in such action by the 
•Board as invalidation of the school's test scores and/or revocation of the 
perpetrator's teaching, counseling, or administrative certificate/s. 

3. Providing Awareness and Instruction for Local School Personnel (1986-1992) 

• Announcements of legislation regarding the implementation and maintenance of 
the Writing Assessment Component of the OSTP are accomplished through 
notices piaced in the Administrative Focus and the Oklahoma Educator in late 
Spring or early Fall semesters. Such notices delineate the legislation and its 
intent; describe the kind of test which will be administered; announce test dates 
and the options offered to school superintendents for choosing available dates; 
specify the population to be tested and provide guidelines for testing Limited 
English Proficient (LEP) students and those with Individualized Education 
Plans (IEP); and announce the Department's plan for offering pretest inservice 
training for all schools. 

• A letter is distributed to all school superintendents early in the Fall semester 
each year. Test dates and procedures are again delineated. Accompanying this 
letter is a survey to obtain data on the number of students to be tested, the 
number of test administrators, the names of building and district test 
coordinators, and the proposed test dates for each school district. 

• Pretest inservice training sessions are conducted for all school districts during 
January and early February each year. Topics for instruction include (1) 
description of the program; (2) implementation procedures and policies— with 
emphasis on maintaining test security; (3) specification of test environmental 
conditions necessary to maintaining test validity; (4) test administration 
procedures; (5) scoring procedures and criteria; (6) descriptions of report formats; 
and (7) procedures for handling test materials (inventory, distribution, coding, 
packaging, mailing, etc.). All inservice participants receive a procedural manual 
which includes specific instructions for building and district test coordinators 
and test administrators. 
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4. Providing Materials and Services 

Test materials are distributed to school superintendents by The Psychological 
Corporation. Such materials include test booklets, teachers' manuals for test 
administration, and the district test coordinators' kits containing all materials and 
instruction for packaging and returning of materials to The Psychological 
Corporation. 

5. Administering the Tests 

Tests are administered on a Tuesday, Wednesday, or Thursday, based on each local 
school superintendent's choice of date selected from three specified weeks of in 
February. Students' test booklets and other materials are returned to The 
Psychological Corporation immediately following administration of the tests. 

6. Scoring and Reporting 

• Students' papers are hand-scored by a minimum of three professional raters per 
paper at The Psychological Corporation's Writing Scoring Center in San 
Antonio, Texas. Part II (pages 15-23) of the Manual for Interpreting describes the 
procedures used to train raters and score Oklahoma's papers. A copy of this 
manual may be obtained from the Student Testing Section, State Department of 
Education (405)521-3341). 

• Results of the tests are distributed to school superintendents in April each year. 
Part III (pages 25-44, Manual for Interpreting) provides information on 
interpreting test results. 



Description of Materials; Testing, Scoring, and Reporting Methods; and Related Services 
A. Materials and Related Services 

The Stanford Writing Assessment Program is a nationally normed direct writing test. 
The format focuses on an exclusively verbal prompt which elicits writing in one of four 
writing modes: narrative, descriptive, expository or persuasive. The descriptive writing 
mode was selected for use in Oklahoma for the 1992 assessment. 

Test materials include (1) a separately sealed page containing the writing prompt; (2) a 
student response booklet which contains one nonscorable page for prewriting activities 
such as planning and/or outlining, and four lined pages for writing; and (3) a test 
administrator's manual containing directions for the examiner to read orally to students. 
Based on HB 1946, enacted in 1990, these materials are not distributed to test 
administrators until the morning of the day testing begins. As the test session begins, 
student test materials are distributed, and each student unseals the writing prompt 
page, listens to directions, and begins writing. 

A District Test Coordinators' Kit is distributed approximately two weeks before the test 
materials are received. This kit contains all materials and instructions necessary for 
efficient management of the testing program in the local school. 
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Pretest inservices are held during the month before testing for every school district in 
the state. State Board Regulations require that at least one representative from each 
school district attend these inservices, and manuals are distributed with oral 
instructions for conducting the assessment according to the standardization practices 
utilized to norm the tests. 

Posttest inservices are conducted throughout the year for all individuals or groups who 
request these services. These sessions focus on interpretation and use of test scores to 
improve instructional programs at the individual, class, building, and district levels. A 
copy of the Manual for Interpreting is provided for each person who attends such 
inservices; two copies are distributed to each school district superintendent with the 
score reports. 

B. Testing Methods 

Standardization procedures call for students to begin the testing session with at least 
two sharpened #2 pencils with erasers. Students are not allowed to have extra paper on 
their desks when the test begins. However, if students need additional paper for 
completing an essay, the teacher is allowed to supply the amount of paper needed upon 
the student's request. 

Materials are distributed, and students complete the school, district and student 
identification information on the front of the response booklet. Students then open the 
sealed prompt The test administrator-following directions uer batim in the official Test 
Administration Manual— reads the prompt orally to the class (two times) while students 
read the prompt silently. Students are then guided through a schedule which permits 
five minutes for prewriting (i.e., planning, outlining); 20 minutes for writing; and 10 
minutes for revising and editing. 

This method of testing is different from that utilized in the MATS Writing Test which 
Oklahoma administered from 1987 through 1991. In using the MAT-6, students were 
given directions and then allowed 20 minutes to complete an impromptu,- writing 
exercise. With the Stanford, students are tested in a manner that applies the writing 
process, the instructional method most strongly advocated in research during the past 
15-20 years. The writing process as an instructional method is advocated in Oklahoma's 
Suggested Learner Outcomes published in 1983 and in the new Expected Learner 
Outcomes published in 1991-92. This method features pre-writing organizational 
strategies, writing, revising, editing procedures and publishing. 

The timed impromptu writing and timed process -writing methods of assessment and 
instruction each serve a distinct purpose which fulfills writing skills needed both in 
school and in the work place. However, the assessment procedure used in the Stanford 
supports a writing method which assists students in learning to organize information 
before writing, and to edit/revise their draft copies to eliminate errors. The learning and 
application of such skills results in improved discourse created for any writing purpose. 

C. Scoring Methods and Reporting Services 

Students' essays are scored by professional writing assessment raters at The 
Psychological Corporation. Scoring of each paper is conducted through use of two 
methods— holistic and analytic. These methods are designed to evaluate different (but 
complimentary) aspects of students' writing skills. 



ii 20 



Each response booklet contains an identification number which is associated with the 
student's name. Before scoring is conducted, the page containing the student's name is 
removed from the remainder of the response booklet. Therefore, raters who scor- j the 
papers are not aware of students' names, gender, or any other identifying information. 

To assure a high level of reliability and validity in scoring students' papers, rater 
reliability stati 'ics are established throughout the scoring prccess. The scores assigned 
papers by each rater are recorded on a computer as each rater completes a set of 25 
papers. Each hour, the rater is checked for accuracy by having another trained rater 
validate the scores being assigned by that person. If a raters quality level begins to fall 
short of an established criterion, the rater ceases his or her work for the day. Use of this 
form of quality control results in strong reliability estimates-a factor which is necessary 
to the production of comparable scores. 

Both holistic and analytic scores are reported to school districts at the individual 
student, school site, and district level. A separate report is provided for the parents or 
guardians of each student. In addition, the State Department of Education is provided 
with group level aggregated scores for school sites, districts, and the state. The State 
Department of Education refrains from collecting or filing any OSTP test data which has 
a student's or teacher's name attached. 

The following scoring methods are used to rate each student's essay: 

1. A holistic scoring rubric which focuses on the overall merit of the paper is utilized to 
rate each essay. The major factors considered in holistic scoring include (a) how well 
the paper is organized (i.e., does it have a good beginning, development, and 
conclusion); (2) how well the concepts introduced are clarified and supported; (3) 
attention to audience in terms of appropriateness of language and content; and (4) 
fluency-the extent to which the writer uses syntax, word choice, and mechanics to 
help the ideas flow smoothly from beginning to end. 

During the standardization (or national norming) of the tests, sets of anchor papers 
were selected. Anchor papers demonstrate characteristics typical of a particular 
score point for a specific prompt. These anchor papers were written by students in 
the grades for which each prompt was being normed and represent examples of the 
quality of writing expected for a paper to be assigned each of the scores in the holistic 
scale. Anchor papers are then utilized for comparisons during the rating of papers to 
assure consistency in assigning the various scores. 

The Stanford Writing Assessment was normed with the use of writing skill standards 
(or expectations) established at four levels of schooling. As one would expect, the 
standards are dependent on the writing mode elicited by the prompt. 

The descriptive mode was used in Oklahoma in 1992 at both 7th and 10th grades. 
The standards used in the holistic scoring of papers written in this mode are 
described as follows: (excerpted from The Manual for Interpreting, The Psychological 
Corporation, 1992). 
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Standard I, Grades 3 and 4 



At the third- and fourth-grade levels, students generally describe several features with 
descriptive details. At the lower score points, both the mentioning of features and the 
addition of some clarifying information are important. At the higher score points, 
students provide more detail and more rigorous organizational strategies. Writers 
earning high scores show attention co the arrangement of items in space. This task can 
be accomplished in one of two ways. Students can either relate features to each other 
("the cup on the table") or to the scene as a whole ("my desk at the front of the classroom). 
Some success at one of these important relational tasks is typical at the higher score 
points. 

Third- and fourth-grade students do not usually include vivid figurative language or 
extremely complicated syntactic structures. Descriptive writing lends itself to the use of 
prepositional phrase modifiers (such as "by the window" or "on the bottom shelf) that 
are strung together. At this stage of development, students generally employ such 
structures, even at the higher score points. The best writers, however, are able to avoid 
monotony by varying the placement of these phrases and by including some adjective 
clauses that enhance the description. For example, a student who writes "the glove that 
my mom bought me" or "the girl who lives next door" is using adjective clause 
description to provide additional information. The best writers will also include some 
specific nouns and single-word modifiers. 



At Grades 5 and 6, students tend to generate more descriptive material than students at 
lower grade levels do. In general, the better writers at Grades 5 and 6 do more than 
merely mention a few features with some detail; some attempt at clarification is typical. 
At the higher score points, the quality of clarification becomes increasingly important. 
For example, at the lower grade levels, students frequently generate lists of colors used 
as predicate adjectives ("The clown's suit is blue, red, and green.") Especially at the 
higher score points, fifth- and sixth-grade papers replace this inventory approach with a 
more specific use of color. In addition, students in the upper elementary grades use some 
complex descriptive strategies, such as condensation, function, or comparison. 

In terms of organization, fifth- and sixth-grade students earning top scores show that 
they have planned their responses from beginning to end. Essays exhibiting overall 
organization do not always contain an explicitly stated plan with varied and precise 
transitions. However, for the most part, such papers progress logically. In relating one 
feature to another or to the scene as a whole, fifth- and sixth-grade writers make more 
systematic use of location than their younger counterparts do. 

Finally, at these grade levels, students' papers exhibit a higher degree of verbal and 
syntactic sophistication. For example, in order to present a clear picture of objectives in 
space, prepositions must be chosen carefully. Topics calling for the creation of a mood 
depend on an exact use of language. Although students' syntax is often heavily 
influenced by the use of phrase modifiers, papers receiving high scores will include 
structures that exhibit some complexity. 



Standard II, Grades 5 and 6 




Standard III, Grades 7 and 8 



Seventh- and eighth-grade students are able to generate a considerable quantity of 
descriptive text. Even at the lower score points, papers exhibit a greater amount of 
information at these grades than at lower grades. For example, in Grades 5 and 6, a 
paper in the middle range includes a few clarifying details for several features. 
However, the same type of response at the seventh- and eighth-grade levels is more 
typical of the lower score range. Although generally students in these grades are 
familiar with nearly half a dozen descriptive strategies (such as number, function, 
condensation, location, and comparison), they are not expected to use all of them. 
However, in high-scoring papers, the strategies chosen are handled well. 

Organizationally, the demands of descriptive writing are also more rigorous at these 
grades than at the elementary levels. A response in the middle range is at least loosely 
organized. Essays receiving the highest scores most often have a well-defined plan that 
is, for the most part, well-executed. It is expected that locations are used systematically 
to relate features to each other and/or the scene as a whole. 

In addition, an increasing number of responses at Grades 7 and 8 exhibit a fairly high 
degree of verbal and syntactic sophistication. A mix of general and specific language is 
common in papers earning lower scores, but superior essays demonstrate precise or vivid 
word choice. A greater proportion of writers at these grade levels are able to generate 
some effective metaphors and similes. Syntax, too, is more complicated at these grade 
levels. Although prepositional phrases are commonly used, quite a few writers achieve a 
degree of sentence variety by including such clause modifiers as "that I bought last 
month" or "which is under your desk." 



Standard IV, Grades 9 Through 12 

At Grades 9 through 12, high-quality material is expected throughout. Vague attempts 
at description of features are not at all helpful at these grade levels. Some vivid or 
specific details are expected. Students earning the highest scores are able to use several 
descriptive strategies (such as location, condensation, and comparison) to present an 
informationally full and unified whole. 

In terms of organization, writers are expected to be more in control of the presentation 
than at other grade levels. Even slightly better than average essays have a clear 
strategy that is, for the most part, carried out. Superior papers exhibit an overall plan in 
which features are related to each other and to the scene as a whole. Some of these 
writers use imaginative organizational strategies. 

Quite a few descriptive pieces at the high school level exhibit an adult-level fluency. A 
considerable number of even the underdeveloped and/or disorganized papers show a 
relative sophistication of language and syntax not found at the lower score points at 
earlier grades. Some of the essays earning higher scores reveal superior linguistic and 
syntactical skill, given the rough-draft nature of the responses. These pieces often use 
highly effective devices such as rhythm, alliteration, and figures of speech to create vivid 
description. Syntactic complexity is often evident, too, as writers experiment with 
fragments, gerunds as subjects, and a variety of adjective and adverb modifiers. 
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Holistic scoring is conducted by two raters who independently read students' essays, each 
assigning a score of one through six. Neither rater knows the score assigned to a given 
paper by another rater. The scores are recorded on a computer and those scores on a 
given paper which are more than one score point apart are identified. Each such 
identified paper is rated by a third person who, by design, is not aware of the scores 
assigned by the first two raters. Scores are again compared, and, at this point, if there 
are not two resulting adjacent scores, the paper is rated by another resolution reader. 
The final assigned raw score of 2-12 represents the sum of the two adjacent scores 
assigned the paper. 

The holistic score scale is utilized to establish normative data (i.e., percentile ranks, 
stanines, and normal curve equivalent scores). It is the score with which an individual's 
or group's performance can be compared with a national and/or state norm. In addition, 
local district normative scores are provided for school districts which enroll more than 
2200 students in grades K-12. 

An analytic scoring rubric is also used to rate each essay. Analytic scores focus on six 
specific categories of writing skill: Ideas and Development; Organization, Unity, and 
Coherence; Word Choice; Sentences and Paragraphs; Grammar and Usage; and 
Mechanics. Raters assign a score of one through four within each category to indicate 
the relative strength of the student's specific writing skills. Figure 1 on page 16 of this 
report displays the specific skill levels associated with the scores assigned within each of 
the analytic score categories. 
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FIGURE I 
ANALYTIC SCORING GUIDELINES 



AREA I: Ideas and Development 




Extensive development of several ideas (or one, important main idea) with extension and 
elaboration on all or most of the points. Look for uniqueness, interest to audience, and strong 
supporting details. Can be exceptional writing or extremely thorough. 

Good development of idea with many details extended and elaborated. Ideas are fairly well 
supported. May be an extensive list with little or no elaboration. 

Adequately supported ideas with some details extended or elaborated. 

Weak ideas minimally supported with little or no extension of details. 



AREA II: Organization, Unity, and Coherence 




Completely organized with smooth flow from one idea to the next through the use of transitions and 
sequencing. Unity is strongly evident with no wandering from the primary theme or plan. 

Fairly well organized with good unity of plan. Some transitions may be used, Little or no 
digression from main ideas or writing mode. 

Small amount of organization. Weak plan that may not be well unified. Ideas may be only 
minimally connected. May frequently wander from expected writing mode. 

Lack of plan or coherence. 



AREA III: Word Choice 




Precise, appropriate, accurate, and specific word choices that convey the correct meaning and 
appeal to the audience. May be vivid and imaginative. 

Word choices that are appropriate and specific. May lack "sparkle," but meaning is clear. 

Fair use of words. May be specific but have little variety. May be simplistic and occasionally 
vague, but generally effective. 

Very simplistic. Meaning may be unclear or inappropriate. 



AREA IV: Sentences and Paragraphs 




Excellent control and formation of sentences. Variety of sentence structure, type, and length 
contribute to fluency and interest, Paragraphs used where appropriate. Few, if any, errors. 

Adequate control of sentence formation. Some mix of sentence types, lengths, and structures. May 
contain a small number of errors that do not interfere with fluency. May have some attempt at 
paragraphing, 

Most sentences constructed correctly. May have little variety in type, length, or structure. May be 
somewhat monotonous or choppy. May contain several errors or lack control. May have no attempt 
at paragraphing. 

Poor sentence structure with many errors that inhibit fluency or clarity. Lacks control. 



AREA V: Grammar and Usage 




Error free or very few errors in approximate proportion to length of paper. 

Good grammar and word usage. Errors that do not detract from overall quality of the paper. 

Fair grammar and usage. Errors may interfere with meaning, May be simplistic. 

Poor grammar and word usage with frequent or serious errors. 



AREA VI: Mechanics 



Error free or very few errors in approximate proportion to the length of the paper, 

Most punctuation, capitalization, and spelling correct. Errors do not interfere with 
communication. 

Some errors in punctuation, capitalization and/or spelling that may interfere somewhat with 
communication. 

Frequent and/or serious errors in mechanics that interfere with communication assigned within 
each of the analytic score categories. 
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Raters who conduct the analytic scoring look for certain characteristics of good writing 
within each of the categories of skills listed above. The rater looks not only for correctness, 
but also for complexity, sophistication, and a variety of writing techniques. The following 
descriptions are excerpted from the Manual for Interpreting (The Psychological Corporation, 
1992) which will be used as Oklahoma's posttest inservice manual: 

• Area I: Ideas and Development 

In this category, readers pay clo * attention to the amount of quality information that 
the student presents. Has the w. . „ar added layers of detail that enrich the discussion of 
key points to the essay? Is there a minimum of bare assertion (statements that receive 
very little, if any, clarification)? Are the ideas interesting, original, and tailored to the 
audience, whether generalized or specific? 

• Area II: Organization, Unity, and Coherence 

Raters scoring papers in this category look for evidence of an overall organizational 
pattern. Is the student in control of the presentation from start to finish? Are 
transitions used effectively to connect ideas both within and between paragraphs? Does 
the writer stay on course by avoiding redundancy and out-of-order material? 

• Area III: Word Choice 

Within the category of Word Choice, raters look for the kind of specific and vivid 
vocabulary that makes writing come alive. Is there a sustained attempt on the students' 
part to use active verbs and precise modifiers that go beyond bland generalities like 
"get," "nice," and "fun"? Does he or she offer on occasion a figure of speech (such as a 
metaphor or simile) that adds a poetic touch to the description? 

• Area IV: Sentences and Paragraphs 

As readers score responses in this category, they look for complex sentence structures 
that enhance meaning. Are sentences structurally complete, smoothly written, and 
clear? Do sentences vary in length and complexity? Is a variety of sentence patterns 
used? 

• AreaV: Grammar and Usage 

In this category, readers look for the correct application of grammatical principles. Does 
the writer use correct subject-verb agreement and logical pronoun reference? Does he or 
she continue to use the tense established in the beginning of the essay? Is every variance 
from the tense appropriate? Does the student show an adequate knowledge of typical 
problems (such as the use of "them" instead of "those")? 

• Area VI: Mechanics 

When evaluating student writing in this category, readers look for consistently correct 
capitalization, punctuation, and spelling. Is a variety of capitalization and punctuation 
skills used correctly? Is the student occasionally successful in using such risk-taking 
techniques as ellipses and dashes to show complexity of thought? Does correct spelling 
enable raters to move smoothly from sentence to sentence? 

Analytic score results are provided for parents and teachers on the score report. Such 
information is useful in providing appropriate instruction for specific writing skill 
improvement, and assists teachers in fulfilling the intent of Oklahoma law. 



17 rsr, 



Holistic and analytic scoring are conducted for different purposes-the former, to provide an 
overall rating of the paper, and the latter, to provide a view of the strengths and weaknesses 
of a student's specific writing skills. While the two method? produce different views of 
writing skill, there is a relationship shared by the two methods. This relationship is 
discussed in section IV of this report. 

The Match with Oklahoma's Suggested Learner Outcomes (1983 ed.) 

As stated previously, one of the major criteria used in selecting an instrument for the 
Writing Assessment Component of the OSTP is the manner in which the test measure 
Oklahoma's Suggested Learner Outcomes (SLOs) for composition. Figures 2 and 3 (pages 
19-20) illustrate the match of SLOs in grades 7 and 10, respectively. Information contained 
in Figures 2 and 3 is explained as follows: 

• Each Suggested Learner Outcome is given in its full text as worded in the SLO Guide. 
Codes in tae extreme left column match the coding of each outcome within the SLO 
Guide. 

• Listed in the column labeled "HOLISTIC" are indications of whether or not each SLO is 
measured by the holistic grading procedure. "Yes" indicates that the SLO was 
measured; "N A" indicates that the SLO could not be measured by this method. 

• Listed in the column labeled "ANALYTIC" are indications of whether or not each SLO 
is measured by the analytic scoring procedure. "Yes" indicates that the SLO was 
measured by this method; "N A" indicates that measuremer" of a given SLO was not 
applicable. 

• Provided in the extreme right column of Figures 2 and 3 is an explanation regarding 
the manner in which measurement of a given SLO is .ddressed by each of the rating 
methods. 
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Understanding and Using the Data 



A. The Comparison of Holistic Scores with National Norms 

Oklahoma's group scores are reported in terms of the AVERAGE sco^e for a specific 
group of students (i.e., building, district, or state). The holistic scores are converted to 
national norms (e.g., National Percentile Ranks, Stanines, and Normal Curve 
Equivalents [NCE] ) to provide a comparison of our students' performance levels with 
those of the national norm group. 

The national norm group is comprised of students whose selection is based on a 
scientific random sampling. The goal of such sampling is to achieve grade level groups, 
with each group representing a cross-section of the United States regarding an 
appropriate balance of geographic areas, socioeconomic levels, ethnic groups, 
urban/rural locations, and other critical factors. 

The norming (or standardization) process involves the administration of appropriate 
test forms to students comprising each grade level national norm group. Each group's 
average raw score on a given test at a particular grade level becomes the "norm" with 
which performance of individuals or groups of students in the same grades can be 
compared. (Example: the average raw score of students in the national norm group is 
represented by the 50th Percentile Rank, the 50th NCE, or a Stanine of 5 on a 
particular test. If Oklahoma's 7th grade students' average score is at the 63rd 
Percentile Rank on a particular test, it can be stated that the "average" 7th grade 
student in Oklahoma is performing at a level that is equal to or better than 63% of the 
students who comprised the national norm group for that test.) 

The Stanford Writing Assessment Program was normed concurrently with the Stanford 
Achievement Test Series, Eighth Edition. Norming took place in the spring and fall of 1988. 
Data were collected on both forms, J and K, at each of the nine levels of the Stanford 
Writing Assessment Program. Equating studies linking forms and levels of the assessment 
were also carried out during standardization. Within a mode only, prompts may be 
compared with one another across grades/levels (e.g., Advanced 1, Grade 7 - TASK 2, Grade 
10) and within a grade/level (e.g., Form J - Form K). The demands on the writer of the 
different modes are great enough that cross-mode comparisons are not recommended. 

Oklahoma's seventh-grade students were administered the Advanced 1 level, Form K of the 
Stanford Writing Assessment, a level specifically designed for the spring of grade 7. The 
tenth-grade students were administered the TASK 2 level, Form K of the Stanford. Form 
K was used at both grade levels because this form is secure. 

The Stanford Writing Assessment offers writing prompts at each level that elicit the four 
major types of writing: Narrative, Descriptive, Expository, and Persuasive. In 1992, 
Oklahoma students at grade 7 and at grade 10 responded to a descriptive writing prompt. 

There are several features of the new Stanford Writing Assessment that distinguish it from 
the MATS Writing Test used previously. Figure 4 (page 22) compares the two writing 
tests. 
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Figure 4 



A COMPARISON OF MAJOR FEATURES OF THE 
MATS Writing Test and the Stanford Waiting Assessment 



FEATURE 

Establishment of 
National Norms 

When Used in Oklahoma 

Tvpe of 

Writing Prompt 

Writing Method 
Measured 



Time Allow ed for Testing 
( not including distribution 
of papers and giving 
directions) 

Writing Modes 
Elicited in the Tests 



Modefs) Used in 
Oklahoma 



MAT-fl Writing Test Stanford Writing Assessment 



Holistic Scoring 



Holistic Sc oring Scale 



Scoring Ranges 
(re: Holistic only) 



1985 

Grade 7: 1988-1991 
Grade 10: 1987-1991 

Combination verbal 
and picture 

Timed impromptu 



Analytic Scoring 



20 minutes 



Each prompt elicits a 
combination of 
narrative and 
descriptive writing. 

Narrative/Descriptive 
in 1987-1991 

Allows for either or 
both modes to be used 



Scored 1 through 8, 
yielding raw scores 
of 2 through 16. 

Use of one scoring 
standard for both 7th 
and 10th grades. 



Four skill areas; four 
performance levels for 
each skill area. 



1988 

Grades 7 and 10: 1992 
Exclusively verbal 



Timed, modified "process" 
writing, with teacher 
instructions guiding students 
through the various steps of the 
writing process. 



35 minutes 



Each prompt is mode-specific; 
available in narrative, descriptive, 
expository, and persuasive modes. 



Exclusively descriptive (1992) 



Mode-specific (any writing not in 
the specified mode elicited is not 
considered for holistic scoring). 

Scored 1 through 6, yielding 
raw scores of 2 through 12. 



Use of different developmental 
writing stage standards-one 
for 7th grade and one for 10th 
grade. Provides for more accurate 
comparisons of writing within 
smaller age/grade ranges. 

Six skill areas; four performance 
levels for each skill area. 
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There are three features of the Stanford Writing Assessment that are new for Oklahoma 
and have a direct impact on the holistic scores and the national norm-referenced scores 
earned by students: 



1. The entirely verbal prompt on the Stanford is scored in a mode-specific way. In 1992, 
Oklahoma students responded to a descriptive prompt. Unlike the MAT-6 Writing Test 
that would score any mode of writing the student produced that was related to the 
prompt,the Stanford requires descriptive writing to a descriptive prompt in order for 
the writing to be scored. This scoring requirement for mode-specific writing means that 
a lengthy descriptive essay which has, for example, long narrative sections, would 
receive a holistic score based only on the descriptive sections of the essay. Mode-specific 
scoring applies in each of the four writing modes tested by the Stanford, 

2. The Stanford Writing Assessment rates each essay on a holistic scale that ranges from 1 
(low) to 6 (high). Two readers independently rate each essay and their scores are added 
together. Thus, the final score for a paper can range from 2-12. This contrasts with the 
MAT-6 Writing Test scoring in which each essay was rated on a scale from one to eight, 
yielding a raw score from two readers that could range from 2-16. The shorter Stanford 
scale is accurate and is more like that used across the nation. It also means that a 
greater range of percentile rank points are tied to each raw score point on the Stanford, 
than on the MAT-6. 

3. On the MATS Writing Test there is one set of rules for judging the holistic (overall 
merit) scores for seventh and tenth grade papers. On the Stanford, there are four sets of 
rules or scoring standards for each type of prompt; one each for primary (grades 3-4), 
intermediate (grades 5-6), junior high school (grades 7-8), and high school (grades 9-12). 
Oklahoma's seventh grade students' papers were judged using the standard appropriate 
for junior high school students. This means that on the Stanford, writing is judged 
against developmentally appropriate standards. 

National norms are available only for comparing students' holistic scores, as this 
scoring technique was the one used in establishing norms during the standardization 
study. The specific criteria used to apply the analytic scoring method to Oklahoma 
students' papers were customized for use in Oklahoma's program. This was 
accomplished to satisfy Oklahoma's request for data to use in (1) identifying students' 
writing skill needs; and (2) planning and implementing appropriate programs to 
accomplish the desired writing skill improvement. Therefore, subsequent discussion of 
"norms" in this report have reference only to students' holistic score averages. 

As explained earlier in this report, two raters independently rated Oklahoma students' 
papers on a holistic scale of 1-6. If the scores given by two raters were more than one 
point apart from each other, a third rater (an experienced reader called a resolution 
rater) scored the paper. Each of the original scores was then compared with the third 
score, and the two closest scores were added to yield the holistic raw score (provided 
that the two closest scores were no more than one point apart). Thus the range of 
possible holistic raw scores was 2-12. 

Tables 1 and 2 on page 24 detail the raw scores, scaled scores and derived norm- 
referenced scores for each grade level tested in Oklahoma. 
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TABLE 1 



Percentile Ranks, Stanines, Normal Curve Equivalents, 
and Scaled Scores Corresponding to the Holistic Raw Scores for the 
Stanford Writing Assessment Program, Level: Advanced I 
Form K, for Spring of 1992, Grade 7* 



HOLISTIC RAW 
SCORES 


PERCENTILE 
RANKS 


STANINES 


NORMAL CURVE 
EQUIVALENTS 


SCALED SCORES 


12 


99 


9 


99.0 


678 


11 


98 


9 


93.3 


667 


10 


95 


8 


84.6 


656 


9 


89 


8 


75.8 


645 


8 


77 


7 


65.6 


634 


7 


60 


6 


55.3 


623 


6 


42 


5 


45.8 


612 


5 


26 


4 


36.5 


601 


4 


14 


3 


27.2 


590 


3 


7 


2 


18.9 


579 


2 


3 


A 


10.4 


567 



TABLE 2 

Percentile Ranks, Stanines, Normal Curve Equivalents, 
and Scaled Scores Corresponding to the Holistic Raw Scores for the 
Stanford Writing Assessment Program, Level: Task 2 
Form K, for Spring of 1992, Grade 10* 



HOLISTIC RAW 
SCORES 


PERCENTILE 
RANKS 


STANINES 


NORMAL CURVE 
EQUIVALENTS 


SCALED SCORES 


12 


99 


9 


99.0 


694 


11 


96 


9 


86.9 


681 


10 


90 


8 


77.0 


669 


9 


77 


7 


65.6 


657 


8 


61 


^6 


55.9 


646 


7 


44 


5 


46.8 


636 


6 


30 


4 


39.0 


626 


5 


19 


3 


31.5 


616 


4 


10 


2 


23.0 


606 


3 


5 


2 


15.4 


596 


2 


2 


1 


6.7 


586 



Administered to Oklahoma's seventh and tenth-grade students in February 1992. 
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Tables 1 and 2 illustrate the effect of spreading a narrow range of scores (such as the 
holistic Raw Score Scale of 2-12) over a larger range of scores (such as the Percentile 
Rank and Normal Curvo Equivalent score scales of 1-99), In applying such 
conversions to real scores, (using the Percentile Rank scale as an example), it becomes 
obvious that the small difference produced by earning one additional Raw Score point 
can make a large difference in the Percentile Rank increase-especially in the "middle" 
of the raw score range. For example, look at the Percentile Rank associated with a Raw 
Score of 7 for Grade 7, and compare it with the Percentile Rank associated with a Raw 
Score of 8. At this point on the scale, a difference of one Raw Score point means a 
difference of 17 Percentile Rank points. Differences in individuals' or groups' norm- 
referenced scores that are in the middle of the scale often appear to be greater than the 
one or two raw score points would indicate. To avoid the harmful effects of such 
distorted implications, we strongly recommend that viewers of test results refrain from 
comparing scores of individual students, school sites, and/or school districts. 

B. The Analytic Scoring Guidelines: Relationship of Analytic to Holistic Scores 

As was described in Figure 1 (page 16), the Analytic Scoring Guidelines included six 
categories of writing skills to be considered: Ideas and Development; Organization, 
Unity and Coherence; Word Choice; Sentences and Paragraphs; Grammar and Usage; 
and Mechanics. Within each of these categories of skills were evaluation criteria 
representing four levels of student performance, with 4 being the highest level. 

Analytic scores are reported (I) at the individual student level by his or her 
performance of 4, 3, 2, or 1 within each of the six categories of skills; and (2) at the group 
level by number and percent of students performing in the 4, 3, 2, and 1 performance 
levels within each skill category. 

The purpose of the analytic scoring technique is to provide data which will assist the 
classroom teacher in determining students' strengths and deficiencies in each of the 
categories of writing skills being measured. These data, used in combination with 
other evidence gathered in regard to students' specific needs, will provide a focus for 
planning writing skill improvement programs for individuals and groups of students. 

Guidelines for appropriate interpretation and use of analytic scores are listed as 
follows: 

• Scores for six skill categories are provided (i.e., Ideas and Development; 
Organization, Unity, and Coherence; Word Choice; Sentences and Paragraphs; 
Grammar and Usage; and Mechanics). During the analytic scoring of Oklahoma's 
papers, each category of skills was considered as independent of the others. 
Therefore, in their use, scores for the six categories are neither to be added to obtain 
a total score nor averaged to determine a mean analytic score. 

• The user of analytic scores is advised to refrain from comparing either the 
individual student's or group's actual performance levels from one skill category to 
another. For example, performance on Sentences and Paragraphs should be judged 
separately to determine (1) the student's actual performance level; (2) the 
deficiencies in skills indicated by that performance level (i.e., the existing need); 
and (3) the instructional techniques, activities, and materials necessary to 
accomplish skill improvement. 
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Failure to examine performance in each category separately could result in 
ignoring a child's need for additional instruction in a given skill area. This error- 
applied at the group instructional planning level-can result in overteaching in one 
skill category at the expense of instruction in another. 

• Holistic and analytic scoring are designed to describe the quality of student writing 
achievement in very different, but complementary ways. The holistic readers make 
a single judgement of the overall merit of the essay. This overall merit judgement 
is based mainly on its organization and its content (development of ideas and 
specific word choice). Spelling, writing mechanics, and grammar can influence the 
holistic score if they make the student's writing difficult to understand. 

The analytic reader looks for evidence of skill and writing proficiency in six specific 
features of the essay. Teachers and parents can use the analytic scores for guidance 
about which aspects of a student's writing are strong and which aspects can be 
improved through instruction. However, because the "whole [holistic] is greater 
than the sum of its parts [analytics]", one will not find a perfect relationship 
between the two scores. 

Nevertheless, there is a positive correlational relationship between the six analytic 
features and the holistic score of students' essays. A study of this relationship by 
The Psychological Corporation obtained the correlations presented in Table 3 
below. 



TABLE 3 

OKLAHOMA SCHOOLTESTING PROGRAM 
DIRECT WRITING ASSESSMENT 



Correlation of Holistic and Analytic Scores for the 
Stanford Writing Assessment Program. Second Edition 



ANALYTIC AREA 


GRADE 7 


GRADE 10 


I. Ideas and Development 


.64 


.57 


II. Organization, Unity and Coherence 


.53 


.48 


III. Word Choice 


.50 


.41 


IV. Sentences and Paragraphs 


.40 


.35 


V. Grammar and Usage 


.36 


.30 


VI. Mechanics 


.38 


.30 



These correlations show that the first three analytic features (Areas I, II, and III) are more 
strongly related to students' holistic scores than the final three analytic features (Areas IV, 
V, and VI). This pattern exists at both the seventh and tenth grades, although the 
magnitude of the correlations is somewhat higher at grade 7. 
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A stronger relationship of analytic areas I, II, and III to the holistic scores was expected. 
These three analytics deal with the content of the essay and its organization and 
development. The high positive correlations validate the contribution of these features to 
the holistic score. It is also clear that none of the first three analytic features totally 
explains the global holistic rating. The final three analytic features (IV, V and VI) are 
more weakly related to the holistic score because these features are of less importance than 
the fir«/t three in explaining the overall merit of the essay. 

Another way of viewing the relationship between holistic and analytic scores is shown in 
Tables 4 and 5 (page 28). At each grade level, we have plotted the average holistic score 
earned by students who received an analytic score of 4, 3, 2, or 1. There is a separate graph 
for each analytic area. (See Figures 5 and 6 on pages 29 and 30). 

The results are consistent, showing: (1) higher holistic scores for students earning analytic 
scores of 4 and 3 compared with those earning analytic scores of 2 and 1; and (2) a wider 
range of average holistic scores within analytic areas I, II, and III, compared with analytic 
areas IV, V and VI. Both results were expected. 

In a smaii number of cases, score reversals did occur. For example, it was possible for a 
student to receive a relatively high holistic score and a low analytic score in one or more 
areas. Conversely, it was possible for a student to receive a middle or low holistic score and 
a high analytic score in one or more areas. 

Each student's essay is unique and must be judged on its individual merits; one 
must be very cautious about generalizations. Nevertheless, when reversals occurred, 
most often one of the following conditions applied: 

High holistic score/low analytic score . These papers had high overall merit (usually the 
paper was written in a manner that addressed the topic and consistently remained "on- 
copic" throughout, complete with supporting details; exhibited good organization and 
development; etc.), but the paper may have been lacking in one or more of the last three 
analytic areas (sentence variety, structure and length; grammar; capitalization, 
punctuation, and spelling). 

Middle or low holistic score/high analytic score . A paper with these characteristics 
could be a simple reversal of th* paper just described-poor content development and 
organization, but excellent use of writing mechanics, grammar, and sentence structure. 
Another possibility is that the paper earned a low holistic score because most of the 
essay was not in the correct mode ("off task"). At the same time, several analytic scores 
could be quite high because analytic scoring considers everything that is written, not 
just the parts writter; in the correct mode. Overall, the holistic and analytic scores of 
the Stanford Writing Assessment show the expected relationship to each other, and give 
the user important information about different qualities of student writing. 
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TABLE 4 

Mean Holistic Raw Score for Each Analytic Score Point 
in Each Analytic Area: Grade 7 



ANALYTIC AREA 


ANALYTIC SCORES 


4 




2 


1 


I. Ideas and Development 


8.82 


7.54 


6.00 


4.37 


II. Organization, Unity and Coherence 


8.79 


7.6S 


6.55 


5.18 


III. Word Choice 


8.65 


7.56 


6.39 


4.56 


IV. Sentences and Paragraphs 


8.22 


7.43 


6.53 


5.80 


V. Grammar and Usage 


7.93 


7.19 


6.39 


5.32 


VI. Mechanics 


8.07 


7.29 


6.51 


5.60 



TABLES 

Mean Holistic Raw Score for Each Analytic Score Point 
in Each Analytic Area: Grade 10 



ANALYTIC AREA 


ANALYTIC SCORES 


4 


3 


2 


1 


I. Ideas and Development 


9.19 


7.81 


6.20 


4.48 


II. Organization, Unity and Coherence 


9.04 


7.95 


6.69 


4.82 


III. Word Choice 


8.96 


7.76 


6.62 


5.30 


IV. Sentences and Paragraphs 


8.64 


7.75 


6.98 


6.22 


V. Grammar and Usage 


8.36 


7.59 


7.00 


6.15 


VI. Mechanics 


8.35 


7.66 


7.05 


6.26 
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Summary of Results and Interpretation of the Data 

A. Document Returns 

Answer documents from all school districts which serve seventh and tenth grade 
students were returned for scoring. State norms were developed based on this return. 

B. Number Enrolled Versus Number Tested 

Not all of the students in grades 7 and 10 were tested due to the following factors: 

• Students who have Individualized Educi..ion Plans (IEPs) pursuant to Public Law 
94-142 were eligible for exemption from OSTP participation (see pages 2-3 of this 
report for a discussion of this issue). If all IEP students are exempted from 
participation in the OSTP, this number can constitute up to 11.1% of the total 
enrollment. 

• Local school district option was offered in regard to testing students who are 
considered as Limited English Proficient (LEP). If all LEP students are exempted 
from participation in the OSTP, this number can constitute up to 1.8% of the total 
enrollment. 

• Makeup tests were not given to students who were absent during the test 
administration. The average daily absence in Oklahoma schools is 5.2%. 

• Student enrollment is reported on September 15th of each school year. These 
figures can decrease measurably for a number of reasons, especially at the high 
school grades. It is apparent that— beginning at 10th grade --dropouts may account 
for many of the school leavers. 

Examples of this enrollment phenomenon are provided as follows: The Fall 1990 
enrollment for Grade 10 was 38,273; the Fall enrollment for this same statewide 
class of students as 11th graders was 35,492-for a total loss of 2,781 ..tudents, or 
7.27% of the class. The actual dropout rate reported for the 10th grade ior the 1990- 
91 school year (i.e., the most recent data available) is 5.83%. 

An illustration with less pronounced results is found in the comparison of Fall 1990 
enrollment at grade 7 (i.e., 43,662) with the Fall 1991 enrollment figures for this 
same class of students in 8th grade (i.e., 43,176). The decrease in enrollment of this 
class is 446 students, or 1.02%. No dropout data are available specifically for 7th 
grade. 

We will not know how many school leavers are accounted for in the difference 
observed in Fall 1991 10th grade enrollment and number of students tested in the 
OSTP Writing Assessment until the Fall 1992 enrollment and dropout figures are 
in. However, a certain percent of this different can be attributable to students who 
are no longer enrolled in Oklahoma schools when the tests are administered. 
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The differences existing in enrollment figures as compared with number of students for 
whom test results are given in this report are indicated as follows: 



Grade Level 



Student Enrollment 
Fall, 1990* 



Number of 
Students Tested** 



% of Student 
Enrollment Tested 



7 
10 



44,792 
40,096 



38,589 
32,611 



86.2% 
81.3% 



♦Source: State Department of Education Accreditation Reports-Fall 1991 
**Source: Maximum number of students reported on the Master List of Test Results 
Summary. (See Appendices B-3 and B-4) 



Summary of State Level Results 

Copies of state level reports will be found in Appendix B (Master List of Test Results 
Summary: 1987-1992) and Appendices C and D (List of Averages-District Level 
Results), Each of these reports is printed in its entirety. The scores for 35 school 
districts in grade 7 and six districts in grade 10 were deleted from the List of Averages 
due to the testing of less than six students. This was done to protect the privacy of those 
students. 

On the whole, the results indicate that Oklahoma's 7th and 10th grade students* 
writing skill performance levels are estimated to be above the national norm. Due to 
the irrelevance of comparing group level holistic and analytic scores, the summary and 
interpretation of these data are discussed categorically as follows: 

1. Normative Data: Holistic Score Summary 

The holistic score average for Oklahoma's 7th grade students who were 
administered the Stanford Writing Assessment was equivalent to a 63rd national 
percentile rank. This means that the average 7th grade student in Oklahoma 
performed at a level which was equivalent to and/or better than 63% of the students 
in the national norm group. 

The average holistic score for Oklahoma's 10th grade students was equivalent to 
the 57th national percentile rank. Thus, the average 10th grade student performed 
at a level that was equal to or better than 57% of the students in the national norm 
group. Table 6 (page 33) provides a comparison of national norms with Oklahoma 
students' performance at grades 7 and 10. 
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TABLE 6 

Comparison of Oklahoma Students' Performance with National Norms- 1992 



Type 

of 
Score 


GRADE 7 


GRADE 10 


National 
Norm 


Oklahoma's 
Performance 


National 
Norm 


Oklahoma's 
Performance 


Scaled Score 


619.00 


624.4 


640 


642.8 


Normal Curve 
Equivalent 


50.00 


57.0 


50.00 


53.5 


Percentile Rank 


50.00 


63 


50.00 


57 


Stanine 


5.00 


6 


5.00 


5 



Each year there are a number of papers that are considered non-scorable due to four 
possible conditions. These conditions, along with codes assigned to each, are described 
as follows: (NS = non-scorable) 



NS-1 = Not scorable because the student refused to write or the student's essay 
had insufficient information, was blank, inappropriate, or a copy of the 
prompt. 

NS-2 = Not scorable because the student's essay was off-topic (i.e., did not 
address the topic of the prompt) or off-task (i.e., was not written in the 
appropriate writing mode). 

NS-3 = Not scorable because the student's essay was written in a foreign 
language. 

NS-4 = Not scorable because the student's essay was illegible or 
incomprehensible. 

The 1992 results indicate that, in Grade 7, there was a total of 217 non-scorable papers; 
this sum for Grade 10 was 223. The number occurring in each of the non-scorable 
categories and the percent of total students tested represented by the sum of non- 
scorable papers received are provided as follows: 



Grade Level 


Non-Scorable 

Category 
Designation 


Number of Papers 
Occurring in this 
Category 


°Io of Papers 
Represented 
by Total 


Grade 7 


NS-1 
NS-2 
NS-3 
NS-4 


45 
170 
0 
2 


.56% 


Grade 10 


NS-1 
NS-2 
NS-3 
NS-4 


67 
154 
1 
1 


.68% 
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Table 7 illustrates the distribution of Oklahoma students' scores by providing the 
number and percent of students whose scores occurred in each quarter of the percentile 
rank scale. A comparison with national norms is offered to illustrate the difference in 
"average" performance and Oklahoma students' performance. ( Appendix A-Percent of 
Scores Under the Normal Curve-illustrates the distribution of "average" group 
performance.) 



TABLE 7 

Comparison of Oklahoma Students' Performance in 
Each Quartile With That Expected Based on the Normal Curve 



NATIONAL NORMS 



Percentile 
Rank 

99 



76 

75 
• 

51 

50 

26 
25 



OKLAHOMA STUDENTS' PERFORMANCE 
Grade 7-1992 Grade 10-1992 




% of Scores 



25% 



#of Scores 



> of Scores # of Scores 



17,000 45% 




25% 



7,427 



19% 





25% 



25% 



10,831 



2,897 



28% 



8% 



% of Scores 



9,499 30% 



.,272 26% 



10,527 33% 



3,867 12% 



An analysis of the holistic scores indicates that the overall writing performance of 
Oklahoma's students appears to be at levels above the average. Our school 
administrators and teachers are to be commended on the provision of instructional 
programs which produce such results. 
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Analytic Score Summary 



Figures 7 and 8 (pages 36 through 39) illustrate the performance levels of 
Oklahoma students within each of the categories of writing skills measured. 
Definitions of performance level criteria are provided for each skill category with 
corresponding percentages of students who scored in each of the respective 
performance levels. 

It appears that the majority of students were able to demonstrate knowledge and 
use of basic writing skills as measured by the Stanford Writing Assessment. In all, 
52<%>-74 < %> of seventh grade students' compositions and 65%-15% of tenth grade 
students' compositions demonstrated capabilities within the two highest levels of 
performance among the various writing skill categories. 
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FIGURE 7 



State Level Results on Analytic Scoring Criteria 
Stanford Writing Assessment, Grade 7-1992 



Area I-Ideas and Development 




100 



50 



100 



Extensive development of several ideas (or one, important main idea) with 
extension and elaboration on ail or most of the^oints. Look for uniqueness, 
interest to audience, and strong supporting details. Can be exceptional 
writing or extremely thorough. 

Good deve loprnent of idea with many details extended and elaborated. Ideas 
are fairly well supported. May be an extensive list with little or no 
elaboration. 

Adequately supported ideas with some details extended or elaborated. 
Weak ideas minimally supported with little or no extension of details. 



4 3 2 1 



Area II-OrganLzation, Unity, and Coherence 



42 40 



4 3 2 1 



■IL 



4 3 2 1 



Completely organized with smooth flow from one idea to the next through 
the use of transitions and sequencing. Unity is strongly evident with no 
wandering from the primary theme or plan. 

Fairly well organized with good unity of plan. Some transitions may be used. 
Little or no digression from main ideas or writing mode. 

Small amount of organization. Weak plan that may not be well unified. 
Ideas may be only minimally connected. May frequently wander from 
expected writing mode. 

Lack of plan or coherence. 



Area Ill-Word Choice 




Precise, appropriate, accurate, and specific word choices that convey the 
correct meaning and appeal to the audience. May be vivid and imaginative. 

Word choices that are appropriate and specific. May lack "sparkle," but 
meaning is clear. 

Fair use of words. May be specific but have little variety. May be simplistic 
and occasionally vague, but generally effective. 

Very simplistic. Meaning may be unclear or inappropriate. 
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FIGURE 7 
(continued) 

State Level Results on Analytic Scoring Criteria 
Stanford Writing Assessment, Grade 7-1992 

Area IV--Sentences and Paragraphs 



100 




4 3 2 1 



100 



Excellent control and formation of sentences. Variety of sentence structure, 
type, and length contribute to fluency and interest. Paragraphs used where 
appropriate. Few, is any, errors. 

Adequate control of sentence formation. Some mix of sentence types, 
lengths, and structures. May contain a small number of errors that do not 
interfere with fluency. May have some attempt at paragraphing. 

Most sentences constructed correctly. May have little variety in type, 
length, or structure. May be somewhat monotonous or choppy. May contain 
several errors or lack control. May have no attempt at paragraphing. 

Poor sentence structure with many errors that inhibit fluency or clarity. 
Lacks control. 



Area V-Grammar and Usage 




Error free or very few errors in approximate proportion to length of paper. 

Good grammar and word usage. Errors that do not detract from overall 
quality of the paper. 

Fair grammar and usage. Errors may interfere with meaning. May be 
simplistic. 

Poor grammar and word usage with frequent or serious errors. 



4 3 2 1 



100 



50 



4 3 2 1 



Area VI«Mechanics 





4 


Error free or very few errors in approximate proportion to the length of the 






paper. 


49 


3 


MoBt punctuation, capitalization, and spelling correct. Errors do not 
interfere with communication. 


19 ■ 25 

HQ 


2 


Some errors in punctuation, capitalization and/or spelling that may interfere 
somewhat with communication. 


■ 11. 


1 


Frequent and/or serious errors in mechanics that interfere with 
communication assigned within each of the analytic score categories, 
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FIGURE 8 

State Level Results on Analytic Scoring Criteria 
Stanford Writing Assessment, Grade 10-1992 



Area I: Ideas and Development 



100 




Extensive development of several ideas (or one, important main idea) with 
extension and elaboration on all or most of the points. Look for uniqueness, 
interest to audience, and strong supporting details. Can be exceptional 
writing or extremely thorough. 

Good development of idea with many details extended and elaborated. Ideas 
are fairly well supported. May be an extensive list with little or no 
elaboration. 

Adequately supported ideas with some details extended or elaborated. 
Weak ideas minimally supported with little or no extension of details. 



4 3 2 1 



Area II: Organization, Unity and Coherence 




Completely organized with smooth flow from one idea to the next through 
the use of transitions and sequencing. Unity is strongly evident with no 
wandering from the primary theme or plan. 

Fairly well organized with good unity of plan. Some transitions may be used. 
Little or no digression from main ideas or writing mode. 

Small amount of organization. Weak plan that may not be well unified. 
Ideas may be only minimally connected. May frequently wander from 
expected writing mode. 

Lack of plan or coherence. 



Area III: Word Choice 




Precise, appropriate, accurate, and specific word choices that convey the 
correct meaning and appeal to the audience. May be vivid and imaginative. 

Word choices that are appropriate and specific. May lack "sparkle/ but 
meaning is clear, 

Fair use of words, May be specific but have little variety. May be simplistic 
and occasionally vague, but generally effective. 

Very simplistic. Meaning maybe unclear or inappropriate. 



4 3 2 1 
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FIGURE 8 
(continued) 

State Level Results on Analytic Scoring Criteria 
Stanford Writing Assessment, Grade 10-1992 



100 



Area IV-Sentences and Paragraphs 



50 



47 




Excellent sOntrol and formation of sentences. Variety of sentence structure, 
type, and length contribute to fluency and interest. Paragraphs used where 
appropriate. Few, if any, errors. 

Adequate control of sentence formation. Some mix of sentence types, 
lengths, and structures. May contain a small number of errors that do not 
interfere with fluency. May have some attempt at paragraphing. 

Most sentences constructed correctly. May have little variety in type, 
length, or structure. May be somewhat monotonous or choppy. May contain 
several errors or lack control. May have no attempt at paragraphing. 

Poor sentence structure with many errors that inhibit fluency or clarity. 
Lacks control 



4 3 2 1 



Area V-Grammar and Usage 



0 





4 


Error free or very few errors in approximate proportion to lengfth of paper. 


47 


3 


Good grammar and word usage. Errors that do not detract from overall 
quality of the paper. 


26 II 23 


2 


Fair grammar and usage. Errors may interfere with meaning. May be 
simplistic. 


ill- 


1 


Poor grammar and word usage with frequent or serious errors. 



4 3 2 1 



Area VI -Mechanics 



100 



50 



45 



25 |9 24 

I H H ■ 

4 3 2 1 



Error free or very few errors in approximate proportion to the length of the 
paper. 

Most punctuation, capitalization, and spelling correct. Errors do not 
interfere with communication. 

Some errors in punctuation, capitalization and/or spelling that may interfere 
somewhat with communication. 

Frequent and/or serious errors in mechanics that interfere with 
communication assigned within each of the analytic score categories. 
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The provision of analytic score data at the individual student level will indicate the 
writing skill area needing the greatest instructional focus and practice. Working together 
with the knowledge base provided by such data, parents and teachers can create 
opportunities for students to become more proficient writers. 

At the classroom and building levels, such data can indicate areas of strength and 
deficiency in specific writing skill instruction based on the number and percent of students 
scoring ateach of the performance levels within each of the skill categories. Such 
knowledge-put into practice-should serve to identify program needs and aid to provide a 
focus for fulfilling these needs. 

Observations 

It appears that, based on the results of the Stanford Writing Assessment, Oklahoma's 7th 
and 10th grade students were able to organize and construct essays in the descriptive 
writing mode in a manner that compares favorably with the performance of the national 
norm group. 

For most students, it takes many years to develop high quality writing skills. This 
requires quality instruction and ample practice in writing. It appears that Oklahoma's 
students are receiving a good measure of both. 

The analytic score scale was neither designed nor normed to establish the expectation that 
25% of all students would score in each of the levels (1-4) within each analytic score area. 
However, the four performance levels do represent varying degrees of writing expertise- 
from 1 (low) to 4 (high). It is therefore desirable for students to earn one of the two highest 
analytic scores (i.e., either a 4 or 3) in each of the six analytic score areas. 

The following table displays Oklahoma students' analytic scores in terms of the combined 
percent of students earning the analytic scores of 4 and 3 in each of the six analytic score 
areas: 

Combined Percent of Students Earning 
Analytic Scores of 4 and 3 



Analytic Score Area 


Grade 7 


Grade 10 


I - Ideas & Development 


66% 


75% 


II - Organization, Unity, & Coherence 


52% 


65% 


III - Word Choice 


59% 


75% 


IV - Sentences & Paragraphs 


59% 


66% 


V - Grammar & Usage 


74% 


73% 


VI - Writing Mechanics 


68% 


70% 



A study of Oklahoma's 1992 writing assessment data indicates that the 10th grade 
students did not perform as well as (1) the 1992 seventh grade group; or (2) their 
predecessors (who took the MATS) when comparing their holistic score means with those 
of their respective national norm groups. However, the analytic score performance of the 
10th grade class shows a greater comparative proficiency in all areas of specific writing 
skill except grammar and usage. 

These results support the recency of Stanford norms as a possible reason for the mean 
holistic score performance level of this year's tenth grade being so much closer to the 
national average than were their predecessors' scores. The newer norms on this test 



40 53 



indicate a higher performance average across the nation in terms of overall writing -i.e., 
the ability to organize thoughts and apply specific writing skills to produce essays that, 
considered as a whole, represent high quality. It appears that Oklahoma's 1992 tenth- 
grade students are being compared with a national average that is more difficult to exceed. 

As illustrated above, the 1992 tenth grade scores exhibit good performance in each of the 
six analytic score areas, meaning that Oklahoma students are applying each of the specific 
writing skills well. The level of writing skills demonstrated by this analytic score 
performance will enable our students to seize the opportunity for improvement in holistic 
scores. 

Directions for the Future 

Oklahoma law requires that the writing assessment data be used to improve instructional 
programs in schools throughout the state. The State Department of Education will continue to 
provide a variety of services to aid with this process. 

A. Posttest Inservice Training Program 

A posttest inservice training program will be offered throughout the 1992-93 school year 
for all teachers, building and district test coordinators, and school administrators serving 
middle, junior high, and secondary level students. Sessions conducted on-site in each 
school district will include the following topics: 

• Interpretation of the Test Data Reports 

• Analysis of Test Results to Determine Needed Changes (methodology) 

• Specific Teaching Techniques and Activities for Writing and Skill Reinforcement 

• Assistance with determining needed changes and innovations in the School 
Improvement Plan 

To prepare for this inservice program, The Psychological Corporation has developed 
materials which are customized to the specific needs of the Writing Assessment 
Component of the OSTP and the follow-up work which will take place at the local school 
level. Such publications include the Manual for Interpreting and additional inservice 
materials which are designed to aid schools in using test data at the local level. These 
materials will be provided for each inservice participant. 

B. Follow-up Services Provided for Schools 

At the request of local school administrators, the Department will provide assistance to 
schools based on the stated needs of each faculty. Since the implementation of the OSTP 
(Fall semester, 1985) the number of such requests has increased. It is anticipated that a 
similar interest in use of the writing assessment data will be exhibited throughout the 
1992-93 school year. 
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APPENDIX A 
Percent of Scores Under the Normal Curve 
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APPENDIX B 

MASTER LIST OF TEST RESULTS SUMMARY: 1987-1992 



Appendix B - 1 
Appendix B - 2 
Appendix B - 3 
Appendix B - 4 



Performance of Oklahoma's Students on the MAT -6 Writing 
Test: Grade 7 for 1988-1991 

Performance of Oklahoma's Students on the MAT -6 Writing 
Test: Grade 10 for 1987-1991 

Performance of Oklahoma's Students on the Stanford 
Writing Assessment: Grade 7 for 1992 

Performance of Oklahoma's Students on the Stanford 
Writing Assessment: Grade 10 for 1992 



NOTE: Oklahoma students' performance levels on the MAT-6 Writing Test for 
1987 through 1991 are not comparable with their 1992 performance on 
the Stanford Writing Assessment, as these tests are not equated* 
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Performance of Oklahoma's Students on the 
MATS Writing Test: Grade 7 
1988-1991 

Holistic Score Results Analytic Score Results 

Performance Levis 



Year 


Percentile Rank 


NCE 


Score Area 


g 

at 


% 


S 2L 


s 


ft 


s 


% 


1988 


57 


5&S9 


I 


4 


14% 


3 


60% 


2 


20% 


1 


6% 








n 


4 


11% 


3 


68% 


2 


18% 


1 


3% 








m 


4 


14% 


3 


43% 


2 


41% 


1 


3% 








IV 


4 


10% 


3 


54% 


2 


29% 


1 


7% 


iqeq 

iuow 




52^3 


i 


4 


15% 


3 


51% 


2 


24% 


1 


9% 








n 


4 


13% 


3 


59% 


2 


23% 


1 


4% 








in 


4 


12% 


3 


55% 


2 


30% 


1 


3% 








IV 


4 


14% 


3 


45% 


2 


30% 


1 


11% 


1990 


m 


53.6 


i 


4 


23% 


3 


47% 


2 


20% 


1 


10% 








n 


4 


19% 


3 


56% 


2 


20% 


1 


5% 








m 


4 


21% 


3 


53% 


2 


23% 


1 


3% 








IV 


4 


17% 


3 


46% 


2 


23% 


1 


14% 


1991 


61 


56.0 


i 


4 


13% 


3 


51% 


2 


27% 


1 


9% 








n 


4 


11% 


3 


57% 


2 


27% 


1 


5% 








m 


4 


10% 


3 


59% 


2 


28% 


1 


3% 








IV 


4 


12% 


3 


46% 


2 


28% 


1 


7% 



Analytic Scoring Guidelines 

Area T-Sentewra Formation 

4 Controlled and varied sentence structure. Few, if any, sentence formation errors. 

3 Basically adequate sentence structure with little variety in length and form. Few sentence formation 
errors. 

2 Some Errors in sentence formation 

1 Frequent and/or serious errors in sentence formation* 

Area II~Word Usage 

4 Vocabulary carefully and imaginatively used. Few, if any, errors in grammar. 

3 Vocabulary acceptable and appropr iate, but somewhat simplistic A few errors in grammar. 

2 Appropriate, but limited vocabulary. Some errors in grammar. 

1 Vocabulary limited and at tunes incorrect or unsuitable. 

Area m.^Ti tent Development 

4 Well-planned content* Clearly elaborated, well-organized* controlled ideas. 

3 Several clear and specific details, with a few problems in either organization or development. 

2 Vague and sketchy details, ofent poorly organized. Problems with repetition of ideas and lack of unity 

1 Weak ideas and random thoughts, poorly planned. Few details included in a very brief response. 

Area IV-Writing Mechanics 

4 No errors in mechanics, or few errors with some attempt at advanced techniques 

3 Few errors in mechanics; little risk-taking. 

2 Many errors in mechanics. 

1 Frequent and/or serious errors in mechanics, interfering with communication. 
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Performance of Oklahoma's Students on the 
MATS Writing Test: Grade 10 
1987-1991s 

Holistic Score Results Analytic Score Results 

PRrformanrp Levels 



Ypar 


Percentile Rank 


NCE 


Score Area 


s 




s 




s 




s 


% 


1987 


74 


633 


I 


4 


32% 


3 


48% 


2 


13% 


1 


7% 








n 


4 


43% 


3 


41% 


2 


11% 


1 


5% 








in 


4 


17% 


3 


60% 


2 


21% 


1 


2% 








IV 


4 


23% 


3 


48% 


2 


20% 


1 


8% 


1988 


69 


60.7 


i 


4 


16% 


3 


60% 


2 


18% 


1 


6% 








n 


4 


15% 


3 


61% 


2 


20% 


1 


4% 








m 


4 


11% 


3 


49% 


2 


38% 


1 


2% 








IV 


4 


9% 


3 


61% 


2 


25% 


1 


6% 


1989 


78 


6&5 


i 


4 


14% 


3 


52% 


2 




t 
l 


y% 








n 


4 


13% 


3 


60% 


2 


22% 


1 


4% 








m 


4 


13% 


3 


64% 


2 


21% 


1 


2% 








IV 


4 


11% 


3 


44% 


2 


32% 


1 


14% 


1990 


72 




i 


4 


25% 


3 


49% 


2 


18% 


1 


8% 








n 


4 


18% 


3 


60% 


2 


18% 


1 


5% 








m 


4 


20% 


3 


60% 


2 


18% 


1 


2% 








IV 


4 


14% 


3 


48% 


2 


24% 


1 


14% 


1991 




66.1 


i 


4 


17% 


3 


65% 


2 


14% 


1 


3% 
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12% 
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1% 
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78% 
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11% 
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1% 








IV 
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17% 
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64% 
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1 5% 


1 


3% 



Analytic Scoring Guidelines 

Area T-Sentenra Formation 

4 Controlled and varied sentence structure. Few, if any, sentence formation errors. 

3 Basically adequate sentence structure with little variety in length and form. Few sentence 
formation errors. 

2 Some Errors in sentence formation* 

1 Frequent and/or serious errors in sentence formation* 

Area Il-Worri Usage 

4 Vocabulary carefully and imaginatively used* Few, if any, errors in grammar. 

3 Vocabulary acceptable and appropriate, but somewhat simplistic A few errors in grammar. 

2 Appropriate, but limited vocabulary. Some errors in grammar. 

1 Vocabulary limited and at times incorrect or unsuitable. 

Area in-Content Devplonment 

4 Well-planned content Clearly elaborated, wellKjrganized, controlled ideas, 

3 Several clear and specific details, with a few problems in either organization or 
development, 

2 Vague and sketchy details, ofent poorly organized* Problems with repetition of ideas and 
lack of unity. 

1 Weak ideas and random thoughts, poorly planned. Few details included in a very brief 
response. 

Area IV»Wrltmg M^hanics 

4 No errors in mechanics, or few errors with some attempt at advanced techniques 

3 Few errors in mechanics; little risk-taking. 

2 Many errors in mechanics. 

1 Frequent and/or serious errors in mechanics, interfering with communication. 

ERIC 
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APPENDIX B-3 

Performance of Oklahoma's Students on the Stanford Writing Assessment: 

Grade 7 for 1992 



T Y M OP WRITING: DESCRIPTIVE 



TOTAL NUMBER TESTED = 38572 1 NON SCQRABLE TOTAL = 217 
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SCORE TYPE 




MEAN 
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MEDIAN 






RAW SCORE 




7.1 




1.7 






7.2 
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18.9 




620.4 
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45.0 
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12 
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0 


678 


99-9 


99.0 


99-9 


99.0 






11 
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2 


667 


98-9 


93.3 


99-9 


99.0 






10 


1963 


5 


£56 


95-8 


84.6 


95-8 


84.6 






9 


5020 


13 


645 


89-8 


75.8 


86-7 


72.8 






8 


9255 


24 


634 


77-7 


65.6 


68-6 


59.9 
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7427 


19 


623 


60-6 


55.3 


46-5 


47.9 






6 


7653 


20 


612 


42-5 


45.8 


26-4 


36.5 
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3146 


8 


601 
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36.5 
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27.2 


5-2 
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24365 
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AREA I 
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N 
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3767 
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15 
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24 


7299 


19 
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19865 


52 


15954 


42 


18621 


49 


16917 


44 


19229 


50 
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11218 


29 
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40 
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38 
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34 
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5 
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1304 
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1343 
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WRITING ASSESSMENT PROGRAM, SECOND EDITION 



TEST OATE: 02/92 
NORMS: SPRING 



GRAOE: 07 

LEVEL: ADVANCED 1 

FORM : K 



TYPE OF WRITING: OESCRIPTCVE 



MASTER LIST OF TEST RESULTS 
FOR 

OKLAHOMA 



HOLISTIC SCORING 

The Holistic Raw Score is the *im of two independent raters using a 6 -point scoring 
scale, with 6 being the highest. The score range > therefore, is Z to 12. 

ANALYTIC SCORING 
The Analytic scores are six separate scores » one for each of six areas: 
Ideas and Development » Organization, Unity, and Coherence; Word Choice \ 
Sentences and Paragraphs; Graeeter and Usage* and Mechanics. A <+-point 
scoring scale is used for each area, with 4 being the highest. 



ABBREVIATIONS USED: 

N, V. * NUN&ER, PERCENT 
NAT'L = NATIONAL 
NCE * NORMAL CURVE EQUIVALENT 
PR = PERCENTILE RANK 



RS 

S 

SS 



= RAW SCORE 

= ST AMINE 

= SCALES SCORE 



FOOTNOTES: 



1 s statistics do not include students with non-scorable essays. 



NA * Analytic score is not available because the student's essay 
was not scorable* 



NA1 = Not available because no derived scores can be reported for 
non-scorable essays. 

NS1 3 Not scorable because the student refused to write or the student's 
essay had insufficient information, was blank, inappropriate, 
or a copy of the prompt* 

NS2 3 Not scorable b ecau s e the student's esse) was off -topic or 
off-task. 



NSS s Not scorable because the student's essay was written in a 
foreign language. 



NS4 = Not scorable because the student's essay was illegible or 
incoaprehens ible . 
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Appendix B -4 

Performance of Oklahoma's Students on the Stanford Writing Assessment: 

Grade 10 for 1992 



TYPE OF WRITING 



DESCRIPTIVE 



TOTAL NUMBER TESTED = 32388 1 



NQN SCORA3LE TOTAL = 



223 
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WRITING ASSESSMENT PROGRAM* SECOND EDITION 



TEST DATE: 02/92 
NORMS: SPRING 



GRADE: 10 
LEVEL: TASK 2 
FORM : K 



MASTER LIST OF TEST RESULTS 
FOR 

OKLAHOMA 



TYPE OF WRITING: DESCRIPTIVE 



HOLISTIC SCORING 

The Holistic Ran Score is the sum of two independent raters using a 6 -point scoring 
scale, with 6 being the highest. The score range, therefore* is 2 to 12. 

ANALYTIC SCORING 
The Analytic scores are six separate scores> one for each of six areas: 
Ideas and Development* Organization* Unity, and Coherence* Word Choice, 
Sentences and Paragraphs \ Gramssr and Usage, and Mechanics. A -point 
scoring scale is used for each area, nith 4 being the highest. 



ABBREVIATIONS USED: 

N, Z a NUMBER, PERCENT 

NAT 1 L = NATIONAL 

NCE * NORMAL CURVE EQUIVALENT 

PR = PERCENTILE RAMC 



RS 
S 

SS 



= RAW SCORE 

= STANINE 

= SCALED SCORE 



FOOTNOTES: 



Statistics do not include students with non-scorable essays. 



NA * Analytic score is not available because the student's essay 
Mas not sc arable. 

NA1 = Not available because no derived scores can be reported for 
non-scorable essays* 

NS1 3 Not scorable because the student refused to write or the student's 
essay had insufficient information, was blank* inappropriate, 
or a copy of the proapt . 

NS2 = Not scorable because the student's essay was off -topic or 
off- task. 



NS3 = Not scorable because the student's essay was written in a 
foreign language* 



NS4 = Not scorable because the student's essay wes illegible or 
inooaprehensible. 
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APPENDIX C 
LIST OF AVERAGES -DISTRICT LEVEL RESULTS: 

Grade 7-1992 
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APPENDIX D 
LIST OF AVERAGES-DISTRICT LEVEL RESULTS: 

Grade 10-1992 



140 



:RIC 



CO 



p 

o 



09 

-63 

>* 

S fl 

** C8 cfl 
8^ C 

*» 0) 3 

e 

C Oj V] 

a> co.S 
Q S* 2 

(« « O 

3 5 2 

0) J 05 

g a? > 

511 

2 so 
— 

^ eft Q) 

fc» ^ 2 
St o 

W «P* mat 

§.§.2 

£ o > . 

;5 a> 0i 

a 0i ©.2 

C 03 w 
© 0 C Q 

a £ a> * 
a S > © 



O 
2 



5 
P 

M 

a 



o 



i 




> 

rH 

tX 



3 

M 



q ui r 
<c > 2 

O J It 



~2 

OS M 

N OC 

M CL 

O (A 



S3 





st 


o 


st 

rH 


LCI 
rH 


Os 
CO 


Nl 

<Sj 


ui 

<NJ 


Nl 
Nl 


h» 


Sf 


o 

CSJ 




rH 


Nl 


CSJ 


OS 

Nl 


rH 




Nl 


Nl 
st 


st 


o 

UI 


rH 
rH 


tn 
ui 


O 

st 


CO 

Nl 


CO 

St 


csj 
U| 


r>. 


s 


2 


3 


CO 

Nl 


U| 
U| 


CO 

st 






CM 

m 


o 

Nl 


o 

CSJ 


o 


cO 

rH 


CSJ 
CSJ 


CO 

rH 


rH 

St 


sO 


rv 


N 
Nl 


st 

CSJ 


'O 


CsJ 




Nl 




rH 


st 


CSJ 


rH 


o 


St 


Nl 
rH 


O 
rH 


St 


o 


a 


Nl 


O 


sO 




o 


st 




st 


a 


o 

rH 


o 

fsj 


o 
o 


Kl 

CSJ 


CSJ 
CSJ 


Nl 
Nl 


o 


rH 

VJ> 


o 

£NJ 


CSJ 
Kl 


st 
CSJ 


Nl 
Nl 


CSJ 


Nl 


rH 




ki 


r*. 
ni 


sO 
sO 


Nl 
st 


rH 
O 


CSJ 

in 


UI 

st 


rH 

St 


rH 

St 


Nl 

CSJ 


Nl 

r*. 


CO 
Nl 


sO 
ut 


CSJ 
U| 


N 

st 


CJS 

Nl 


3 




> 

csj 


CSJ 
U| 


*t 
CSJ 


Nl 
Nl 


o 


H 

CSJ 


o 

Nl 


sO 

CSJ 


CSJ 

ui 


sO 
rH 




CO 

CSJ 


CsJ 


rH 


CSJ 


CSJ 


CM 




rH 


H 
H 


o 


ui 


o 


Nl 


Nl 


o 


r- 


O 


o 


Nl 


o 


Nl 


Nl 


o 


Nl 




st 


csj 


st 


U) 

CSJ 


o 
o 


rH 
rH 


CO 

rH 


o 

rH 


U| 
rH 


Nl 




s 


CO 


Nt 
Nl 


rH 

CSJ 


o 


o 

CsJ 


□ 


Nl 


CJS 
Nl 


CSJ 
sO 


co 

Nl 


rH 
O 


O 
U| 


CsJ 
U| 


CO 

Nl 




Sf 


5 


Nl 


3 


sO 


CsJ 
Nl 


CO 
st 


pH 

U| 


u < 

(0 ui 


> 
H 

eg 


Nl 

st 


csj 


Nl 




ui 

Nl 


CsJ 


st 


CJS 
rH 


CSJ 


CSJ 


s0 
CSJ 


,t 

st 


SO 


sO 

CSJ 


CSJ 
U| 


r>. 

rH 


OC 

< 


rH 


ui 


CJ 
rH 


Nl 


o 


st 


co 


U| 


Uj 


o 


a 


CO 


o 


o 


rH 

CsJ 


o 


sO 


u 

H 


st 


o 


CO 


O 
rH 


sO 
rH 


o 

rH 


Nl 
rH 


CO 


ui 

rH 


CSJ 


CsJ 


rH 






sO 


o 


o 

rH 


I- > 


Kt 


Nl 
sO 


CSJ 


CO 
ut 


st 
CO 


st 
f>. 


CSJ 

so 


CJS 
sO 


s 


CO 

SO 


R 




OS 






CO 

>t 


r>. 
ui 


> CO 


H 
M 

M CM 


ni 

HI 


o 

CSJ 


<0 

CSJ 


o 


sO 
rH 


U| 

CSJ 


csj 








rH 


CO 


Cjs 


rH 

st 


CO 
st 


rH 
Nl 


< 


«H 


st 


o 


m 


o 


O 


o 


o 


o 


o 


O 


o 


o 


o 


CsJ 


Kl 


CSJ 


z 




































< 


st 


O 


CO 


CO 


CsJ 
<t 


o 

rH 


U| 


Nl 
CsJ 


st 


sD 
rH 


o 

st 


U| 
rH 


o 


rH 






rH 




Nl 


CO 

st 


J 


s 


st 


h. 
ui 


Nl 
sfl 


rH 

St 


sO 

ui 


rH 
sO 


o 

sO 


sO 
ui 


5 


ON 
Nl 


CO 

Nl 


CO 
so 


CO 
sT 




M 
M 

CsJ 


OS 
Nl 


CO 
Kl 


o 

st 


rH 
rH 


Kl 

CSJ 


O 
Nl 


sO 
Nl 


rH 

st 


Nl 

CSJ 


o 


Os 

CSJ 


CSJ 
UI 






CSJ 


Nl 






M 


o 


ui 


O 


rH 


CSJ 


O 


O 


O 


o 


O 


o 


o 


CSJ 
rH 


Nl 


Nl 




st 


O 


o 

CSJ 


o 

rH 


Nl 


CO 

rH 


CSJ 


CO 
CM 


rH 
rH 


sf> 
CSJ 


Nl 
Nl 


o 

CsJ 


CO 


rH 

CsJ 


Nl 


Nl 
rH 


rH 
rH 




Nl 


rH 

sO 


o 


NJ 

in 


Nl 
sO 


rH 
W 


CO 
h- 


s 


O 


8 


O 
sO 


o 

SO 


CSJ 

h» 


Nl 

N 


Nl 

ut 


~n 

U| 


tn 

sO 




H 

CsJ 


M 
M 


a 

rH 


Nl 
Nl 


a 


O 

to 


o 

CSJ 


CO 

rH 


CJN 

rH 


SO 
rH 


h. 


oo 

rH 


o 

CsJ 


sO 


CsJ 
Nl 


SO 
rH 


CsJ 
CSJ 




H 




O 


ui 


a 


rH 


O 


O 


O 


O 


O 


CsJ 


o 


o 


CsJ 
rH 


sO 

rH 


CsJ 



h a 



5 H uj 

r S3 



< oc 
Z a. 



o 

CO 



H h» OS 



rH O 

ui m 



r>. co 

Kl st 



st 

I 

O 



U| 
I 

CsJ 
U| 



U| 

I 

CsJ 
U| 



ui 

I 

Nl 
1A 



CO 



st 
CSJ 

ui 



UI 

st 

ut 



ui 
I 

ui 



ut in 

i i 

OS rH 

U| tfl 



UI 
I 

OS 

U| 



as? 



H N M 



O 

sO 

st 



sO 
OS 

st 



h rs» 



st 
rH 

ui 



o 

CO 



sf* sO OS 

so os 



st 

CO 



r>.r>.rs. 



o 

UI 



o 

st 



r- 

QC 
UJ 

co 

-4 

o 
o 



a 

g 

u 

o 
u 



u 



1 



o 
u 




UJ 

3 S 



o 



3 



z 

to 

UJ UJ 

a x 
o t— 
u 

00 u» 

o 

y w 

H rH 



10 



O 
Z 



a 
> 

L 

a 

2 

tfl 

H-» 
"ft 



> 

(0 



H-» 



I 

g 

9 



O 



o 

o if 



c 5 



o 



s) 

:RIC 



(r 




Q 

2 







iH 

M 


Ul 


ro 


ui 


Kl 

tfl 


O 


CO 


o 


OS 


tN* 
CM 


Kl 
K* 


CM 


KJ 


2 


S 


o 




Ml 


h» 

st 


Kl 

CsJ 


iT> 
si» 


sO 
Kl 


Kl 
KS 


o 

Kl 


CM 
mD 


o 


O 

r-- 


un 
«a- 


sO 
UI 


vO 
Kl 


UI 
ui 


sO 


S 


o 

't 




M 

^ M 


sO 
rH 


UI 
<H 


rH 
rH 


UI 

Ui 


st 
r-t 


a 
\o 


Kl 

CM 


o 
o 

rH 


rH 


CM 


rH 
rH 


sO 
Kl 


rH 
rH 


2 


rH 


Kl 




rH 


UI 


CO 


st 


UI 


o 


o 


CO 


O 


st 


O 


o 


o 


eg 


o 


M 
tH 


M9 




st 


st 

Kl 


CT» 
sO 


fs. 
OJ 


(0 
fH 


ui 

si* 


o 
CsJ 


<0 


O 


CM 
CM 


CK 


CM 


K- 


CSJ 




Kl 


CM 




Kl 


Na- 
si- 


rH 
Kl 


iH 
U| 


sD 
Kl 


ui 

st 


o 

CM 


rH 
Kl 


CM 


Kl 
st 


UI 
•d* 


CO 


rH 

st 


s 


Uj 


O 


»H 




> 

CM 


CO 


O 


O 

CsJ 


LCI 
st 


o 

iH 


O 
sO 


st 


iH 
h« 


CM 


NO 

Kl 


O 


CM 


ON 


2 


rH 


SO 
CM 


(A 


rH 


st 


o 


Kl 


o 


o 


o 


CO 


o 


ON 


ON 


o 


o 


eg 


o 


O 


(0 


ui 
or 


st 


h» 

csj 


ui 
CO 


h» 

CsJ 


st 
rH 


rH 

st 


o 
csj 


CO 


o 


CM 


CO 


rH 


eg 


eg 


o 


vO 


CO 
rH 


o 


Kl 


r*. 

<t 


ui 

rH 


on 

st 


OS 
U) 


f>. 

'J* 


o 

st 


M> 
st 


OA 


Kl 

st 


s 


Kj 


Kl 


eg 


eg 
CO 


rH 
Kl 


UI 
^ 


u < 

V) UI 


> 
rH 

CsJ 


rH 

CM 


o 


o 

CsJ 


r^. 

CM 


CM 
rH 


o 

S* 


CM 


rv 
ui 


O 


vO 


sO 


CM 


CM 


2 


>^ 


CM 


or 
< 


rH 


U| 


o 


st 


o 


O 


o 


ON 
rH 


v* 

rH 


st 


o 


o 


o 


Kl 


o 


ON 
rH 


CO 


o 

M 


<f 


CsJ 
CsJ 


UI 
M 


o 

CM 


U! 


ui 

Kl 


o 

rH 


O 


o 


o 

Kl 


o 


H 


ui 


CM 


o 


M» 


r>. 


K > 


Kl 


sO 

ui 


h- 
h- 


ui 


iH 
sT 


st 


o 

CM 


Kl 
f»« 


st 
rH 


Kl 

st 


st 
sO 


r*» 

M) 




si- 

M> 


rH 


iH 


u> 


> CO 
-J 


M 
M 

m eg 


rH 

CU 


£0 


© 

CSJ 


rv 

csj 


CO 

r^ 


O 


rH 


ON 

CM 


sO 
CM 


sO 

Kl 


eg 

CM 


Kl 

eg 


vf 


Os 


NO 


CM 


< 

z 


rH 


H 


o 


Kl 




o 


o 


CO 


ui 


O 


O 


o 


O 


O 


o 


o 


rH 




st 


Kl 
CsJ 


rH 
Kl 


rH 


CO 
rH 


Kl 
Kl 


o 

rH 


o 


O 


<M 
CM 


o 


o 


Kl 

eg 


CM 
rH 


o 


M) 


Kl 




Kl 


CO 

s» 


CSj 
sO 


CSJ 

ui 


UI 
UI 


os 
st 


O 
Kl 


OS 
sO 


o 


Kl 
<t 


Kl 

rN» 


s© 


ui 

st 


st 
sO 


CO 
rH 


rH 
CO 


sO 

ui 




esj 


CO 

CM 


CO 


CsJ 


f>. 
CM 


CO 

rH 


o 

sO 


fv 
CM 


. o 
o 


O 
Kl 


CM 


Kl 
Kl 


is. 

CM 


<r 

CM 


CM 
CO 


M 


o 




rH 


CsJ 


o 


st 


O 


o 


o 


v* 


fH 

o 




o 


O 


ui 


O 


o 


O 


rH 




<f 


O 
Kl 


rH 
Kl 


<r 

rH 


ON 


rH 

va- 


o 

rH 


O 


o 


UI 

Kl 


o 


eg 
eg 


<i" 

rH 


rH 
rH 


«7> 


o 

UI 


rH 




Kl 


rH 

ui 


ON 
sO 


sO 


ut 
ut 


in 


o 
ui 


Kl 

r>. 


st 
rH 


3 


CM 

CO 


st 
st 


<T> 

ui 


CO 
ui 


Kl 
f>« 


o 
ui 


Kl 

ui 




M 

CSJ 


M 


O 


Kl 
rH 


sO 
Kl 


rH 


o 

st 


Kl 

CM 


SO 

CO 


N 
iH 


CO 

rH 


Kl 
Kl 


Kl 

eg 


eg 

Kl 


CO 

rH 


o 


r«» 

CM 




H 


CsJ 


o 


st 


O 


O 


o 


st 


o 


O 


o 


O 


UI 


o 


O 


o 


eg 



?5 
o « 
H h si 



out 

2> 3 
Ui o 



u2 

JS M 



UJ 






ON 


ui 


eg 


rH 




Kl 


eg 


Ui 


sO 


Ui 


o 


CO 


Kl 


sO 


MEAN 
STAT 
NCE 


ui 
ui 


M> 

ui 


Kl 

ui 


UI 


Kl 

NO 


U| 


o 
ui 


Kl 
st 


ui 

st 


ON 
U| 


sjr 
Kl 


ON 
st 


h» 

st 


CO 
Kl 


eg 
ui 


N0 

xt 


UJ 


g> 


nO 


ui 


vO 


NO 


U) 


U) 


<t 


U| 

t 


sO 

1 


st 

1 


U| 

» 


U| 

» 


st 
1 


Ui 

t 


ui 
i 


STAT 
PR-S 


o 

M> 


CM 

NO 


t 

K« 
UI 


vO 


1 

V* 

f>. 


1 

CO 
UI 


> 

rH 
U| 


1 

CO 
K| 


rH 

st 


N 
SO 


Kl 

CM 


ON 
st 


sT 
st 


o 
Hi 


st 
U| 


st 
s* 




rH 


s* 


si" 


u\ 


a 


h» 


s* 


NO 


st 


iH 


O 


rH 


O 


Kl 


o 


st 


5 r- UJ 

IU < u 

r z z 


s 


ON 
UI 


*Q 
U| 


ON 
UI 


u* 

M> 


SO 

U| 


Kl 
UI 


sj* 


CJN 

Nl- 


rH 
sO 


rH 

st 


Kl 
U| 


o 

UI 


st 
st 


ui 

U| 


o 

Ui 


-J 


NO 


M> 


SO 


SO 


sO 


SO 


UI 


U| 


UI 


SO 


st 

1 


U| 

t 


U| 

» 


st 

1 


Ui 

t 


ui 

t 


NAT' 
PR-S 


t 

UI 

NO 


1 

r^. 

Ml 


1 

CM 
sO 


M> 


1 

N0 
P«- 


Kl 

NO 


1 

o 

UI 


t 

sO 

sT 


1 

c* 

sfr 


a 

fs» 


st 
Kl 


sO 
UI 


CM 
UI 


ON 
Kl 


ON 

Ui 


rH 

ut 


5 in 

UJ CO 


rH 

5 

V0 


CJN 
S0 


s* 

NO 
st 
sO 


CO 

0% 
M> 


o 

sO 
UI 


a 

NO 

st 
sO 


r«. 

CM 
sj" 
sO 


rH 

sO 
Kl 
sO 


o 

CO 
Kl 
M> 


rv 

rH 
UI 

sO 


CM 
CO 

eg 

sO 


CM 

CH 
st 
sO 


ON 

OS 
Kl 
sO 


U| 

CM 
Kl 
sO 


K* 

i 

SO 


st 

CJN 

Kl 

NO 


3» 

u^ at 


rH 

CO 


eg 

CO 


o 

CO 


Kl 
CO 


CO 

CO 


O 

CO 


sO 

hi 


O 

fs! 


CM 
fsl 


ui 
CO 


CM 
sO 


Ui 


Kl 

hi 


sO 
sO 


CO 


Kl 

hi 



I 

CO 



< 

3 



X 

UJ 
M 
> 



a 



U. rH rH CC 
3 < < < 



s a 



ui Q 



ac o 

UJ (A 

tH as 

c? 3 



z -I 

tH Ui 
UJ ui 

or x 

O r- 

O 

a 

u 

rH IT) 



-J (A 

Ht I 



ON 

I 

Ui 



z 

3 

UJ 

§ 

a. 



> 



H-f 



H-» 

Q 



s 

~3 



0 
O 
CO 



CO 



CD i 

- c; 



a. «- 

o = 

a < 



:RLC 



Q 

2 



3 



o 



a 

s 

rH 

r- 
M 

s 



a ui £ 

«< > a 

a ui o 

UJ _J u. 



«3 

Cfs M 

x ac 

CSJ Q» 
O 01 

UJ 





st 


o 

CM 


O 
r-i 


CSJ 

r-t 


rH 


co 

CsJ 


rH 
rH 


rH 


CM 


M 
rH 






s 


CsJ 


HI 


CH 


3 






$ 


OS 

St 


sO 
HI 


OS 
vt 


M» 

Ht 


a 


sO 


st 


Sf 


5 




CO 


fs. 
st 




s 


rH 

>» 




M 

> 

CM 


O 

CM 


rH 

>r 


sO 

ro 




CsJ 




CSJ 


CM 




Ht 


st 


o 

rH 


CO 

CM 


OS 

CsJ 


O 

CM 


vO 
rH 




rH 




o 


sO 
rH 






Uj 


o 


O 


rH 


O 


CsJ 


o 


HI 


o 


rH 


CM 










CO 


CSJ 


rH 




rH 


CM 


sO 


h» 


CO 


o 

CM 


HI 
HI 


CO 

HI 


CO 
CM 


CM 
st 




HI 


5 




O 

st 


s 




CM 


fv 
sO 


st 


M> 


o 

CO 


U| 
U| 


o 

h- 


r*. 

st 


CM 
U| 


ON 

ui 


CM 
Vf 




csj 


•st 

rH 


3 




s 


CsJ 


vp 
st 


CM 
Csj 


(f\ 
CM 


o 

tA 


HI 
pH 


r- 

Hl 


O 


ON 


rH 


st 


>t 


01 




o 


HI 


2 


a 


H 


CO 


o 


o 


st 


o 


O 


o 


o 


o 


o 


CsJ 


LU 

a 


st 








rH 


60 


o 


M) 


rH 


sO 


HI 
K» 


HI 


o 


CM 
CM 


r«» 

CM 


o 

CM 


U| 
HI 


o 


HI 




<0 

H\ 


3 




st 


s 




ON 
st 


U| 

ui 


o 

st 


O 

ui 


o 

OS 


r*. 


r*. 
ui 


rH 
U| 


CsJ 
st 


u < 

01 UJ 


> 
►H 

CM 


HI 

CM 


5 


CM 


(Si 


HI 


st 


CM 


CO 
HI 


sO 

CM 


o 

CM 


HI 
HI 


o 

rH 


M 
HI 


sO 
rH 


ON 

CM 


rH 

CM 


ac 
< 


rH 


O 


HI 


ON 


ui 


vt 
rH 


tv 


O 


CM 


rH 


rv 


CO 


O 


O 


a 


rH 


HI 


u 

M 


st 


CJN 


ui 


m 


ut 


HI 


O 


o 


f»» 


sO 








rH 


rH 




CM 


t- >■ 


HI 


Kl 

a) 


SO 

Lfl 


HI 
tfl 


o 
h- 


vt 


(JN 
HI 




ui 


ON 


HI 
U| 


M> 


ON 






M> 


SO 


> CO 


M 

> M 

m eg 


on 


00 
HI 


h- 
Hl 


CsJ 

csj 


CsJ 


ui 


sO 


sO 
HI 


st 


st 




3 




CM 


CM 


rH 


< 




o 


a 


>t 


HI 


<0 




o 


O 


CM 


o 


CM 


a 


HI 


o 


rH 


CsJ 


z 


rH 


































< 


st 


ON 


U) 


HI 


CO 


HI 


O 


O 


o 

rH 


rH 


CM 






CSJ 


CM 


hw 


OS 
CM 




HI 


41 

st 


(Js 
UI 


Cfs 
UI 


csj 

vO 


HI 
HI 


sO 






2 


fv 

<r 




Q 

HI 


CO 

s 


rH 

sD 


o 


Kl 

ut 




M 

CsJ 


s0 

st 


HI 
HI 


csj 
HI 


CH 


sO 

ui 


!S 




S 


CM 


fv 

CM 


^t 


o 
rv 


ON 

rH 


sO 
rH 


HI 
U| 


rH 




rH 


O 


HI 


r*. 


ui 


CO 




o 


rH 


vO 


O 


CM 


o 


O 


O 


G 


rH 






HS 
CM 


ut 




st 
rH 




o 


i>» 

rH 


st 
rH 


rH 
rH 


o 


st 


o 


UI 
CM 


UI 
CM 


CsJ 


U| 
HI 




HI 


NO 

sO 


ON 

ut 


a 

sO 


ON 

UI 


CsJ 
tv 




CM 


M» 
UI 


O 

st 


o 

SO 


<t 
sO 


o 

fv 


HI 
U| 


ON 

U| 


CO 
sO 


O 
UI 




H 

CsJ 


ON 


V0 
HI 


CsJ 
Hi 


<SJ 

CsJ 


ON 
rH 


CJN 


rH 
rH 


co 

CM 


HI 

vt 


o 

st 


rH 

H> 


o 

CM 


CM 
CM 


sO 
rH 


ON 

CM 


CsJ 
rH 




H 


o 


o 


rH 


ui 


sO 




O 


CM 


SO 


o 


rH 


o 

rH 


O 


O 


rH 


CsJ 








UI 


(JN 




ui 


CsJ 


r. 


o 


HI 


9> 


rH 


o 


rH 




CO 


rs. 


UI 


00 


0) 


5 < uj 

UJ ^ o 

r oi 2 


ui 


rH 


rH 
U| 


CsJ 
st 


sO 

st 


CM 


CM 
Ut 


CM 

st 


CM 
U| 


3 


O 
U| 


w 

st 


rH 

st 


CM 
U| 


CsJ 
st 


sO 
UI 




































ORE 


Ui 

K 01 
< » 

h- a 

01 a 


1 

ui 


T 

st 
HI 


ui 

1 

HI 

ui 


T 

sO 
Ml 


U| 

i 


HI 
1 

sO 
rH 


U| 
1 


f 

fN. 

HI 


ui 

» 


UI 
1 

sO 
st 


ui 
o 

U| 


T 

<0 
HI 


st 

1 

ui 

HI 


ui 

1 

U| 

ui 


st 

1 

sO 
HI 


sO 

1 

HI 

sO 


a 


_i 


r*» 


N> 


CO 


rH 


ut 


^ 


rH 


Si 


M» 


eo 


U| 


U| 


sO 


ui 




st 




3 >- uj 
S3* 


S0 

ui 


sO 

«t 


3 


>» 


o 
ui 


sO 


U| 
U| 


r*. 

st 


Ui 
U| 


rH 
U| 


HI 
U) 


CO 

st 


sO 
>t 


ui 
ut 


r*. 


ON 

ut 


a 


.J 


S0 




ui 


ui 


U| 


<t 


SO 


UI 


sO 

1 


UI 
l 


UI 
1 


ui 

1 


UI 
1 


sO 

t 


ui 
i 


N0 

I 


M 


- c/> 

h- 1 


1 

HI 

s9 


i 

j 


ON 

UI 


i 


1 

rH 

UI 


sO 
csj 


o 

sC 


i 

sO 
st 


rH 

sO 


HI 
UI 


r*. 

U) 


r*. 

st 


$ 


o 

sO 


ut 


r-. 

sO 


V) 




































tH 




S0 


es 


HI 


CM 


sO 


sO 




CSJ 


CM 


rH 


0> 


HI 


sO 


CM 


CP 


sO 


_J 

o 




S0 

st 

N0 


•t 

HI 
sO 


| 

sO 


ui 

HI 
sO 


HI 
VO 


M 
sO 


i 

sO 


sO 
HI 
sO 


ui 

st 
sO 


rH 

st 
sO 


Csj 
st 
sO 


H"l 
sO 


st 
HI 
sO 


i 

sO 


ui 

HI 
sO 


OS 

st 
x0 


X 


z 


O 


CO 




ON 


Hi 


sO 


CO 


O 


O 


U| 


sO 


rH 


CO 




O 


CM 



CO 
ZD 



s 

5 



2 5 



UJ 

CO 



2 § 



hC 

S 



22 

M 

§5 

M UJ 

(71 

UJ UJ 

02 X 

a h- 

c?) u. 
o 

o 

rH W 



s 

Si 

UJ 

8 



> 



> 
U 



2 

I 
S 

a 

-a 

u 

0 



£2 



.a 

- o 



ERIC 




> 
3 



P 



UJ -J 

Q UJ Z 

< > 5 

at uj o 

Uj u. 



S QC 
CM Q. 
O W 



< 

O 



C/3 UJ 



vt 


HI 

CM 


a 

rH 


rH 
rH 


CM 


CO 
CM 


rH 


HI 




rH 


c5 


Kl 




3 


si- 


M> 

HI 


HI 


Ul 
rH 


<T 
CsJ 


rH 

U| 


rH 

U) 


o 

st 


r*. 

st 


CM 


rH 

si" 


C* 
U| 


<t 




3 






HI 


M 

> 

CM 


sO 

st 


rH 

Ul 


h- 

CsJ 


HI 

CsJ 


U| 
CsJ 


CsJ 
st 


CM 


rH 

si* 


CM 


UJ 


CO 


rH 


rH 


U| 




rH 


Ul 
rH 


sO 
rH 


rH 
rH 


<f 


sO 


O 


HI 


sO 


o 


st 


O 




CM 


o 


HI 


st 


rH 
HI 


HI 


rH 
rH 


rH 
CM 


o 

HI 


sO 
rH 


CM 






CM 


Jo 




s 


3? 


f>, 
HI 


HI 


co 

HI 


Ul 
HI 


CM 
U| 


rH 

Ul 


<* 
>t 


j* 


Ul 


U| 

st 


s 


CO 


2 


sO 


CM 


HI 
U| 


O 


> 

CM 


rH 

HI 


sD 

st 


CO 
CsJ 


U| 
CsJ 


Ul 
rH 


rH 
rH 


HI 
rH 


HI 

st 


CsJ 
rH 


st 
HI 


o 


Uj 


CM 


Ul 


o 


rH 


O 


sO 
rH 




HI 


SO 


O 


sO 


Ul 


HI 


U| 


o 


rH 


o 


o 


HI 


st 


rH 
HI 


Ul 


h. 


r*» 

r-i 


o 

CM 


CM 
HI 


CM 
^l 


U| 


HI 


HI 


HI 

rH 


Ul 
rH 


Kl 
st 


5 


CM 


HI 


rH 
HI 


CM 
*\ 


rH 
U| 


o 

U| 


sO 

*t 


S 

in 


Ul 

st 


sO 


st 


st 
HI 


Csi 
st 


sO 

st 


CM 


CM 


HI 
st 


> 
M 

CM 


Ul 
rH 


CO 
r* 


HI 
HI 


sO 
CM 


sO 

CM 


rH 
rH 


<r» 

rH 


HI 
CsJ 


co 

rH 


CM 
U| 


sO 
si" 


CO 

CM 


CM 


rH 


CM 




HI 

CM 


Ul 

CM 


Cfs 


h- 


CO 


O 


HI 


Cs 


sO 


rH 
rH 


o 


O 
rH 


O 


U| 


sO 


st 


HI 

CM 


Ul 


<0 


HI 
rH 


f*» 
rH 


Ul 


O 
rH 






U) 


CM 




rH 


CM 


rH 


HI 


Ul 


<0 


HI 
U| 




O 

sO 


N. 
Kl 


CO 


<M 
U| 


s 


rH 


r-H 




Ul 


2 




M 
M 

rH CM 


HI 

CM 


sO 
Ul 


Ifl 
HI 


OS 
rH 


HI 

CM 


Ul 


sO 


HI 


st 

CsJ 


Ul 


CO 


CM 


CM 


sO 


CO 


rH 


O 


CM 


U| 


rH 


rH 


o 


o 


O 


O 


U| 


o 


O 


O 


O 


HI 


<t 


rH 

HI 


HI 




O 
rH 


HI 

CM 


rH 


rH 


H 










rH 


s 


HI 
CM 


HI 


in 

rH 


OJ 


rH 

st 


U| 


HI 
U| 




st 


ON 
HI 


sO 


Uj 


tn 


R 


sO 


rH 


s 


M 
M 

CM 


<t 
Ul 


<r> 

U| 


st 


HI 
HI 


HI 

CM 


sO 


s 


HI 

st 


rH 


HI 


CM 


CO 


rH 


rH 


CM 


rH 


O 


o 

rH 


U| 


Hi 


rH 


O 


O 


CM 


O 


st 


O 


o 


o 


o 


CM 


st 


CO 


HI 


rH 
rH 


HI 
rH 


rH 


U| 


HI 
rH 








rH 


2 


H 


st 


CM 


Hi 


o 

Ml 


CSJ 

U| 


CO 
U| 


o 

Ul 


U| 
U| 


O 
CO 


N. 

rs 


f>. 
U* 


sO 

r>. 


HI 

r>. 


U| 


eg 
h» 


CJ\ 

f>. 


CM 
sf 


U| 
Ul 


M 

CM 


HI 
CM 


r- 
hi 


M> 

CM 


sO 

c« 


CM 
rH 


U| 


a 

rH 


S3 
HI 


rH 

CM 


sO 
rH 


CO 


o 

rH 


h. 


sO 
rH 


CO 
rH 


rH 


O 


<0 


U| 


rH 


CM 


o 


O 


Ul 


O 


CsJ 


o 


O 


o 


O 







UJ 




HI 


rH 


rH 


U| 


O 


CM 


CM 


O 


sO 


CM 


rH 


st 


sO 


ON 


c/i 


5 5 ui 

UJ r> U 

r cfl z 


5 


rH 

st 


sO 
st 


O 

st 


rH 
sO 


? 


CM 
sO 


HI 
»t 


CM 
sO 


rH 
U| 


HI 

sO 


O 
U| 


CM 
st 


cn 

st 


r«. 

st 


































ORE 


STATE 
PR'S 


Ul 

1 

CO 

sT 


t 

«t 
M 


Ul 

1 

HI 
•t 


U| 
1 

CO 


sO 
1 

rH 
hw 


U| 

J 

CO 

sf 


M> 
1 

cm 

N. 


«t 

( 

N. 
M 


sO 
1 

HI 


ui 
1 

HI 
U| 


sO 
1 

>t 
N. 


ui 
1 

o 

U| 


t 

sO 
HI 


Ul 

1 

c* 

st 


ui 

sO 
>t 






st 


Ul 


^ 


<* 


HI 


CM 


st 


a 


st 


st 




CM 


ON 


sO 


CO 


V) 


5 h uj 

111 4 u 
X Z 2 


CM 
U| 


sO 
st 


O 
U| 


CSJ 

U| 


HI 
sO 


CM 
U| 


HI 
sO 


co 

st 


M> 


i 


sO 


HI 
U| 


sO 
st 


HI 
U| 


rH 
U| 


CJ 


-J . 


U| 


U| 


U| 


U| 


<J> 


Ul 


M) 


m 


sO 


U| 


sO 


U| 
i 


U| 

1 


in 

i 


U| 
1 


M 


NAT 1 
PR-S 


1 

U| 
U| 


1 

HI 

>* 


1 

rH 
U| 


sO 
U| 


1 


1 

$ 


1 


sO 

Nf 


1 

Ul 

fs» 


S 


s9 

N. 


M) 
Ul 


i 


Ul 


HI 
U| 


C/l 


































rH 




CO 


HI 


rH 


rH 


O 


m 


HI 


HI 


HI 


rH 


CO 




st 


sO 


h. 


-J 
O 


uj 


rH 

sO 


st 
HI 
sO 


0* 
HI 

sO 


<SI 

$ 


2 

sO 


rH 

^t 
sO 


2 

vO 


SO 
HI 
vO 


Ul 

in 

sO 


3 

sO 


Ul 

ui 

s» 


CM 
st 
sO 


Ul 
HI 
sfl 


CSJ 
st 
sO 


a 

st 
sO 


X 




Ul 


CO 


CM 


Ul 


vO 


Ul 




O 


CO 


CO 


CO 


MB 


C* 


ui 


st 




uj c5 

z 


N 


M> 




N 


CO 




CD 




CO 




CO 




sO 




hi 



































3 

5 




(0 V) 



CO 

CO 



I- UJ 
rH 

S3 

rH 

CI 

111 Ul 

cr x 

8- 

173 U. 
O 

u 



—J C/J 

3£ 



o 
o 

> Si 
41 

O £= 



5.5 



C < 



:RLC 



(? 




2 



a lu X. 

< > 2 

a uj o 

U J Ik 



\ at 

CM Q. 

o w 



ft 



CO sO 

H 8 




5 

-J 
< 

t 

2 





Nt 


<t 
cm 


Kl 
K- 


N- 


\o 

CM 


rH 
rH 


ON 


H 
rH 


M5 
CM 


•i* 


rH 


CM 


CM 


O 


KJ 


CM 


O 




Kl 


w 


tv 


CM 
Ifl 


rv 
tfl 


CM 

tv 


Ifl 


CM 

h» 


^2 


r*. 


rH 

Ifl 


Ifl 
tfl 


MJ 

"J" 




S 


JO 






H 
> 

CM 


Ifl 

CM 


o 


CO 
Kl 


CK 


rv 

rH 


CM 
Kl 


h- 

rH 


Kl 

CM 


O 


iH 
Kl 


CO 
rH 


Kl 

CM 


Kl 

\f 


CM 


CM 


fv 




rH 


M) 


o 

CM 


Kl 




o 




o 


Kl 


M> 


vt 


O 


CO 


ON 
f— ( 


>t 


Ifl 


o 






CM 


o 

MJ 


%t 
rH 


o 

Kl 


rH 
rH 


M3 
rH 


MJ 


si* 
CM 


rH 


rH 
rH 


CM 
Nt 


Kl 
CM 


a 

H 


K* 

rH 


Kl 
CM 


o 

M> 




Nl 


>t 


Kl 
Kl 


sD 
M> 


ifl 
Kl 


rH 

M> 


ON 


M> 
Ifl 


KJ 
tfl 


rH 

st 


O 
MJ 


tN. 

Nt 


CO 
Kl 


Kl 
Kl 


o 
Ifl 


Nt 
Nt 


Kl 
Kl 




CM 


CM 


fv. 


rH 

CM 


vO 
CM 


CO 

CM 


\* 
Kl 


CN 
HI 


a 

CM 


CO 
rH 


Ifl 

CM 


rH 
iH 


rH 


Kl 


ON 

CM 


rH 

CM 


fv 






ifl 


o 


o 


a 


O 


rH 


O 


CM 


O 


st 


o 


CO 


rH 


Nt 


CM 


o 




rH 


































UJ 

□c 


Nt 


MJ 
rH 


fv 
MJ 


rv 


r** 

rH 


M) 


ON 


o 


CM 


rH 

<r 


h. 


.H 
CM 


CO 


a 


o 


ON 


tfl 


o 


Kl 


M) 
st 


fv 

CM 


en 
tfl 


rH 

M> 


iv 

MJ 


rH 

Nt 


CM 

h» 


M> 


CM 


O 
ifl 


rH 


**i 


KJ 


CM 


CM 


fv 

CM 


u < 

W uj 


> 

CM 


rH 


rv 




CM 
CM 


CM 
CM 


rH 


CO 

CM 


MJ 

CM 


CM 


On 


MJ 
lH 


rH 

Kl 


Kl 
M^ 


rH 


CM 


CM 


(X 


rH 


CO 


o 


O 


O 


M) 


CO 


O 


nT 


M> 


>!" 


Kl 


CO 




CO 


CM 


O 


«« 




































u 

H 


Nt 


H 


rv 


h» 


ON 


O 


CM 


o 


a 

CM 


CM 


rH 


Kl 


O 


Ifl 


O 


ON 


Kl 
Ifl 


h- > 


K» 


CO 
ifl 


rv 
co 


M) 
CO 


Kl 
CO 


ON 
CO 


O 


Kl 
CO 


O 

Ifl 


r>. 


CM 

CO 


ifl 


MJ 


rH 


rH 


MJ 


rs. 


> a) 


H 
KH 

M CM 


CO 

CM 


fv 


tv 


0 


rH 
rH 


tfl 


fv 

rH 


o 

CM 


rH 




CM 


rH 


CJ 


CM 




o 


< 




CM 


o 


O 


o 


O 


<t 


o 


rH 


O 


o 


o 


o 


O 


O 


o 


o 


2 


rH 


































< 




Kl 
•H 


H 


N. 


rH 


o 


CM 


M> 


rv 

•H 


ON 

CM 


rH 


Ifl 


o 


<r 


O 


CM 


rO 




M 


M> 

vt 


h- 

>t 


CM 
M) 


rH 

M> 


Kl 
CO 


Nt 
Kl 


rH 

M> 


o 
tfl 


ON 
Ifl 


ON 

h» 


rH 

K» 


Kl 

CM 


rH 


o 

CM 


CO 
Kl 


fv 

Nt 




H 

CM 


MJ 
Kl 


a 


rH 

Kl 


M> 

CM 


r-» 

rH 


iv 
ifl 


K» 
Kl 


rH 

K\ 


CM 
rH 


ON 

rH 


CM 


f>. 


rH 


iH 


vt 


o 




rH 


Ifl 


o 


o 


o 


o 




O 


CM 


o 


o 


o 


O 


o 


o 


CM 


o 




Nt 


fs. 
rH 


<M 


o 

rH 


M) 
CM 


o 


tfl 


M> 


0> 
fH 


Ifl 

Kl 


Nt 


Kl 


o 


ON 

rH 


o 


Kl 
rH 


Kl 

rv 




Kl 


Kl 
ifl 


K» 


M» 


Ifl 

M> 


CM 
fv 


ON 

Kl 


M> 
Ifl 


JN- 
Ifl 


rv 

Nt 


MJ 

r>. 


MJ 
MJ 


o 
o 

rH 


r>. 

MJ 


CO 
Kl 


in 

MJ 


fN». 

CM 




M 

CM 


fV 

CM 


O 


Nt 

rH 


a» 


CO 

CM 


O 

s* 


0> 
Kl 


Kl 

CM 


CO 
rH 


On 
fH 


CM 
Kl 


O 


Nt 

rH 


N1 
MJ 


rH 

CM 


O 




H 




O 


O 


o 


O 


M) 


O 


rH 


O 


o 


O 


o 


o 


o 


O 


o 








UJ 




o 


OK 


CM 


Ifl 


>t 


Ifl 


M) 


CM 


N 


o 


N 




Ifl 


ON 


fNN. 


(A 


3&.U 
uj r- U 

X Cfl 2 


M) 

>t 


M» 
>t 


3 


M> 

Ifl 


Kl 
st 


a 


O 


ifl 

■cr 


MJ 

N* 


MJ 
Ifl 


MJ 
Kl 


MJ 
Ifl 


o 

IA 


CM 

Nt 


M> 

NT 


rH 

rv 




































iu 


UJ 

« f 

(0 a 


Ifl 


tfl 


Ifl 


M» 


vt 




t 


ifl 

i 


Ifl 

1 


M> 
1 


t 


M> 
1 


MJ 

1 


t 


in 
i 


r*. 
i 


o 


1 

Kl 

Nt 


Kl 

*t 


I 

CO 

*t 


1 

CM 
M» 


l 

CO 
Kl 


CM 
Kl 


Kl 
Kl 


CM 


Kl 

Nt 


CM 
MJ 


Ifl 

CM 


Kl 
MJ 


h. 

MJ 


MJ 
Kl 


Nt 
Nt 


ifl 

CO 


u 


.j 


M) 


*t 


rH 


Kl 


O 


M> 


rH 


ON 


CO 


Ifl 


rH 




O 


CM 


ON 


ON 


t/1 


x - 

< l- UJ 


O 
Ifl 


o 
ifl 


Kl 
Ifl 


3 


s 


tfl 


M) 


o* 


o 
Ifl 


CO 

Ifl 


CM 

St 


CO 
Ifl 


rH 

MJ 


Nt 


o 

Ifl 


rH 

rv 


u 




Ifl 


ifl 


m 


MY 


Ifl 


Ifl 


Ifl 


Ifl 


Ifl 


Ml 




M> 

I 


MJ 

I 


Ifl 

1 


tfl 

1 


rv 
i 


H 

w 


- CJ> 
r— I 

22 


1 

rH 
Ifl 


i 

rH 
Ifl 


i 

M) 
Ifl 


I 

Ifl 
M) 


1 

M> 
>t 


1 

CM 


1 

Kl 


t 

o 
tfl 


t 

CM 
Ifl 


i 

MJ 
MJ 


Ifl 

Kl 


MJ 
MJ 


O 
h. 


Ifl 
Nt 


<M 
Ifl 


ifl 

CO 


</l 




































w 




Kl 


rH 


CM 


Ifl 


IV 


M> 


CM 


M> 


rH 




st 


CM 


Ifl 


fN* 


CJX 


MJ 


-J 
o 


X 


CK 
Kl 
M> 


On 

Kl 
M> 


CM 
Nt 
Ml 


3 

M) 


M> 

M) 


Kl 
Kl 
M) 


Kl 
M> 


S 

M) 


o 

>t 

MJ 


3 

MJ 


C* 

CM 
MJ 


On 

xt 

MJ 


rH 

Ifl 
MJ 


Ifl 

Kl 

MJ 


Kl 
MJ 


Kl 
MJ 
MJ 


X 




Kl 


Kl 


Ifl 


CM 


rH 




CO 


CM 




CM 


Kl 


CM 


>t 


O 


Kl 


tfl 




uj a 
X 


tv 


fv 


iv 


CO 


N 


M> 


MJ 






CO 


MJ 


CO 


CO 


IV 


IV 


ON 





































Si 



cr 

uj ui 
> > 



5 3 

> > 



13 2 

Z < 



CO 

uj ui 
CC X 
O r- 
U 

(/) uu 

o 

U 

JH « 
h- M 
>* 



rv: 



T3 

a 

2 

a 



> 

2 

(S 
> 
0 



J3 



9 

H* 

■s 

*r* 
t- 

8- 

a 
it 

H* 
It 



2 

S 



o 



: < 
r*0 



CD 



^ S3 
-2 C 



- C3 

>? 

o — 
u < 



9 

:RLC 



